Testing and automation. These two are key to ensuring high quality of software releases.<\/p>\n\n\n\n
Ever since I worked briefly in the team at MySQL AB that is responsible for creating the binary (and source) packages of MySQL releases, I have had the vision of a fully automated release procedure. Whenever someone pushes a new commit to the release branch revision control tree, the continuous integration test<\/a> framework should kick in and do all the steps needed for producing release packages:<\/p>\n\n\n\n To do this efficiently, clearly the use of virtual machines is needed. This weekend I played with KVM<\/a> and Buildbot<\/a>, and managed to set up a proof-of-concept of this that I am really pleased with.<\/p>\n\n\n\n There are lots of options for virtualisation these days, including KVM, Xen, VirtualBox, and Vmware. I use KVM, and I really like it. The integration into the distributions is excellent ( I started by installing a basic ubuntu Jaunty server in a virtual machine:<\/p>\n\n\n\n I use the user mode network stack with port forwarding for ssh access. This allows to run kvm without root privileges, avoids any need to manage different MAC addresses, avoids the need for routing or configuring interfaces, etc.<\/p>\n\n\n\n Using the Incidentally, I did this using remote X over an SSH connection. This works fine, no need for physical access to the host server. After installation we will run the virtual machine without a graphic console, but it was just easier to use the stock Ubuntu installer than trying to find a way to install over the emulated serial port.<\/p>\n\n\n\n Next I did some basic preparation to make the installed virtual machine work well for command line and script usage. However, the amount of extra packages installed is kept to a minimum to get proper testing against unwanted dependencies.<\/p>\n\n\n\n I Installed ssh server for remote access. I then set it up to use the serial console (as we will be running kvm in -nographic mode). To get a login prompt on serial port 0, create To get the kernel to output its boot log to the serial port, edit the kernel line in Next, we need a user account inside the virtual machine that we can use from the outside with passwordless login and Then, in the host create an SSH public\/private key pair without passphrase:<\/p>\n\n\n\n Copy the resulting Now we should be able to test that things work:<\/p>\n\n\n\n We now have the basis for scripting actions against the virtual machine: We can start up the guest from the command line (and shutdown with The qcow2 virtual hard disk image format used by qemu (and kvm) has a very powerful feature, activated with the This creates a new image We use this to create a virtual machine that we will use to produce the source tarball from bzr sources. This needs installing bzr and some development packages (compilers etc).<\/p>\n\n\n\n Note the very<\/em> nice Another virtual machine image is set up for building the binary packages (this does not need bzr):<\/p>\n\n\n\n (with a bit more planning, I could have cloned Finally, a third image for testing installation:<\/p>\n\n\n\n I will be testing a bintar package install, so create the mysql user and group:<\/p>\n\n\n\n With these preparations, we should be ready to put the pieces together:<\/p>\n\n\n\n For MariaDB, we use Buildbot for continuous integration testing. The Pushbuild system I developed at MySQL was never released publicly, and in any case it is better to use a general tool like Buildbot that is widely used and maintained by a large community.<\/p>\n\n\n\n I have been very satisfied with Buildbot. It has its quirks and bugs, but we can fix those over time (and have fixed a number of them already, as well as added extra features we needed). I think Buildbot has all of the right ideas for doing serious continuous integration testing. As I read in some presentation, running the builds and tests is the easy part. The hard part is providing the information and tools needed by developers to fix problems that are found by testing. Fixing these problems is what it is all about, after all, not just producing pretty status reports.<\/p>\n\n\n\n First, I installed a buildbot slave on the host machine:<\/p>\n\n\n\n Then I set up an account for this in the Buildbot master, and configured the builder.<\/p>\n\n\n\n With the above preparation, configuring the build is just setting up the proper shell commands to be run against the slave, although it is of course a bit more involved than for a normal I included the complete config in all detail at the end of this post, but here are the main points.<\/p>\n\n\n\n The After starting the virtual machine, we wait for boot to have completed by This gets the base name of the source tarball into a Buildbot build property<\/em>, an essential feature of Buildbot for more advanced usage. We will need this name in the following build steps (the name depends on the version of the MariaDB server code).<\/p>\n\n\n\n We copy out the generated source tarball (we will need it in the next buildstep, which runs in a different virtual machine). We then shutdown this guest, and wait for it to finish with another Here (and in the following install step), we use And that’s it! Full config details below, but it is basically the same, just with different commands run in the different steps. The result is a builder that fully automatically tests build and install on real machines with the correct setup, 100% repeatable between builds.<\/p>\n\n\n\n This is just a quick proof of concept, but I think all of the essential ingredients are in there. I am hoping that in the not too distant future we will be using something like this regularly to check MariaDB release builds, which should be very good for getting ensuring both the quality and efficiency of future MariaDB releases!<\/p>\n\n\n\n Testing and automation. These two are key to ensuring high quality of software releases. Ever since I worked briefly in the team at MySQL AB that is responsible for creating the binary (and source) packages of MySQL releases, I have had the vision of a fully automated release procedure. Whenever someone pushes a new commit… Continue reading Building MariaDB\/MySQL with Buildbot and KVM<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[28,35,4,6,38],"_links":{"self":[{"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/posts\/106"}],"collection":[{"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/comments?post=106"}],"version-history":[{"count":1,"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/posts\/106\/revisions"}],"predecessor-version":[{"id":107,"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/posts\/106\/revisions\/107"}],"wp:attachment":[{"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/media?parent=106"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/categories?post=106"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/knielsen-hq.org\/w\/wp-json\/wp\/v2\/tags?post=106"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}KVM<\/h2>\n\n\n\n
sudo apt-get install kvm<\/code> and you’re up and running). The interface is powerful and flexible, and at the same time really simple to learn and use. Just a couple of commands with man pages, like it should be in a Unix system. Basically, it just works!<\/p>\n\n\n\n qemu-img create -f qcow2 vm-jaunty-i386-base.qcow2 8G\n kvm -m 2047 -hda vm-jaunty-i386-base.qcow2 -cdrom ubuntu-9.04-server-i386.iso \\\n -boot d -smp 2 -cpu qemu32,-nx -net nic,model=virtio -net user -redir tcp:2222::22\n<\/pre>\n\n\n\n
virtio<\/code> network driver greatly improved throughput for me when copying things into and out of the virtual machine. The -cpu qemu32,-nx<\/code> (disable “No eXecute” support) is needed in this case due to some bug or incompatibility, or the installation hangs upon reboot. As usual Google is your friend in cases like this:<\/p>\n\n\n\nInitial setup<\/h2>\n\n\n\n
\/etc\/event.d\/ttyS0:<\/code><\/p>\n\n\n\n start on stopped rc2\n start on stopped rc3\n start on stopped rc4\n start on stopped rc5\n stop on runlevel 0\n stop on runlevel 1\n stop on runlevel 6\n respawn\n exec \/sbin\/getty 115200 ttyS0\n<\/code><\/pre>\n\n\n\n\/boot\/grub\/menu.lst<\/code>, removing quiet splash<\/code> and adding console=ttyS0,115200n8 console=tty0<\/code>. To get Grub to use the serial port, add these lines to \/boot\/grub\/menu.lst<\/code>:<\/p>\n\n\n\n serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n terminal --timeout=3 serial console\n<\/pre>\n\n\n\n
sudo<\/code> access. Inside the guest, create the account and grant passwordless sudo<\/code>:<\/p>\n\n\n\n sudo adduser --disabled-password buildbot\n sudo adduser buildbot sudo\n sudo visudo\n # uncomment `%sudo ALL=NOPASSWD: ALL'\n<\/pre>\n\n\n\n
ssh-keygen -t dsa\n<\/pre>\n\n\n\n
~\/.ssh\/id_dsa.pub<\/code> from the host into ~\/.ssh\/authorized_keys<\/code> in the guest.<\/p>\n\n\n\n kvm_pid_2222' ; exec kvm -m 2047 -hda \/kvm\/vms\/vm-jaunty-i386-makedist.qcow2 \\\n -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic \\\n -net nic,model=virtio -net user\n # We should get a login prompt in the terminal window\n ssh -p 2222 buildbot@127.0.0.1 'sudo id'\n # We should get root access without login or sudo asking for password.\n<\/pre>\n\n\n\n
kill<\/code> from the host or sudo shutdown -h now<\/code> from the guest). And we can run commands inside the guest using ssh -p 2222 buildbot@127.0.0.1<\/code>. The next step is to create variants of this base virtual installation for the different purposes we need.<\/p>\n\n\n\nqemu-img create -b base_image.qcow2<\/em><\/h2>\n\n\n\n
-b<\/code> option of qemu-img create<\/code>:<\/p>\n\n\n\n qemu-img create -b vm-jaunty-i386-base.qcow2 -f qcow2 vm-jaunty-i386-makedist.qcow2\n<\/pre>\n\n\n\n
vm-jaunty-i386-makedist.qcow2<\/code>, which is initially a clone of the base image vm-jaunty-i386-base.qcow2<\/code> that takes up (almost) no extra space. But as we use this new image, changes are added in the new image (copy-on-write), without modifying the original base image. This allows painless mass cloning and modification of virtual machines without having to re-install, and without taking up unnecessary extra disk space and I\/O for copying images.<\/p>\n\n\n\n sudo apt-get install bzr\n sudo apt-get build-dep mysql-5.1-server\n<\/pre>\n\n\n\n
build-dep<\/code> feature of apt-get, it actually installs a ton of packages needed to build the MySQL server (and MariaDB has the save dependencies). I also copied in an existing shared bzr repository; this is not strictly necessary, but saves a very painful initial cloing of the entire MariaDB repository from Launchpad (bzr is just painfully slow on source trees of the size of MariaDB\/MySQL):<\/p>\n\n\n\n scp -rp -P 2222 .bzr buildbot@127.0.0.1:\n<\/pre>\n\n\n\n
qemu-img create -b vm-jaunty-i386-base.qcow2 -f qcow2 vm-jaunty-i386-build.qcow2\n<\/pre>\n\n\n\n
-makedist<\/code> from -build<\/code>; now I just repeated the install of mysql-server-5.1<\/code> dependencies, but not the bzr install. Also, a refinement wouldbe to setup the -build<\/code> guest without autotools and bison, to check that build is possible without those installed).<\/p>\n\n\n\n qemu-img create -b vm-jaunty-i386-base.qcow2 -f qcow2 vm-jaunty-i386-install.qcow2\n<\/pre>\n\n\n\n
sudo adduser --system --group mysql\n<\/pre>\n\n\n\n
Buildbot<\/h2>\n\n\n\n
sudo apt-get install buildbot\n sudo addgroup buildbot kvm # To allow buildbot to run kvm\n sudo -u buildbot buildbot create-slave --usepty=0 \/var\/lib\/buildbot\/maria-slave \\\n hasky.askmonty.org:9989 knielsen-kvm-x86 <password>\n<\/pre>\n\n\n\n
configure+make<\/code>. I really like the simplicity of this. Basically, after initial preparation of the KVM images, there is very little setup required on the buildbot slave host, it is all just normal shell commands configured on the master. Of course going forward we can refine some of this and maybe put some of it into generic scripts called from the main config, but for a proof-of-concept I think it is brilliant that one can see exactly which commands are run.<\/p>\n\n\n\nf_kvm_jaunty_x86.addStep(Compile(\n logfiles={\"kernel\": \"kernel_2222.log\"},\n command=[\"sh\", \"-c\", \"\"\"\nkill -9 \"$(cat kvm_pid_2222)\"\n(exec sh -c \"echo \\$\\$ > 'kvm_pid_2222' ; exec kvm -m 2047 \\\n -hda \/kvm\/vms\/vm-jaunty-i386-makedist.qcow2 -redir 'tcp:2222::22' -boot c \\\n -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio -net user\" \\\n <\/dev\/null >kernel_2222.log 2>&1) &\nsleep 15\nwhile : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done\nssh -p 2222 buildbot@127.0.0.1 'mkdir -p buildbot && cd buildbot && \\\n rm -Rf build && bzr co \"lp:~maria-captains\/maria\/mariadb-5.1-knielsen\" build && \\\n cd build && BUILD\/compile-dist && make dist && \\\\\n mv \"$(make show-dist-name).tar.gz\" ..'\n\"\"\"]))\n<\/pre>\n\n\n\nkill<\/code> command removes any previous left-over kvm process (better safe than sorry). We run kvm in the backgroud, getting the console output through a log file. Note that redirecting the kvm output is necessary, as the buildstep will wait for all processes to close the stdout before considering the buildstep done.<\/p>\n\n\n\n
checking for successful ssh connection in the while<\/code> loop. Once it
is ready, we send the commands to build the source tarball into the guest
using ssh.
<\/p>\n\n\n\nf_kvm_jaunty_x86.addStep(SetProperty(\n property=\"distname\",\n command=[\"ssh\", \"-p\", \"2222\", \"buildbot@127.0.0.1\", \"cd buildbot\/build && make show-dist-name\"],\n ))\n<\/pre>\n\n\n\n
f_kvm_jaunty_x86.addStep(ShellCommand(\n command=[\"sh\", \"-c\", WithProperties(\"\"\"\nscp -P 2222 buildbot@127.0.0.1:buildbot\/%(distname)s.tar.gz .\nssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'\nwhile : ; do sleep 5; kill -0 \"$(cat kvm_pid_2222)\" || break; done\nrm -f kvm_pid_2222\n\"\"\")],))\n<\/pre>\n\n\n\n
while<\/code> loop. Note the use of WithProperties<\/code> to interpolate the source tarball name obtained in the previous build step.<\/p>\n\n\n\nf_kvm_jaunty_x86.addStep(Compile(\n command=[\"sh\", \"-c\", WithProperties(\"\"\"\nqemu-img create -b \/kvm\/vms\/vm-jaunty-i386-build.qcow2 -f qcow2 vm-tmp-2222.qcow2\nkill -9 \"$(cat kvm_pid_2222)\"\n(exec sh -c \"echo \\$\\$ > 'kvm_pid_2222' ; exec kvm -m 2047 -hda vm-tmp-2222.qcow2 \\\n -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio \\\n -net user\" <\/dev\/null >>kernel_2222.log 2>&1) &\n# ...\nssh -p 2222 buildbot@127.0.0.1 'rm -Rf buildbot && mkdir buildbot'\nscp -P 2222 %(distname)s.tar.gz buildbot@127.0.0.1:buildbot\/\nssh -p 2222 buildbot@127.0.0.1 'cd buildbot && tar zxf %(distname)s.tar.gz && \\\n cd %(distname)s && .\/configure ...'\n# ...\n\"\"\")],))\n<\/pre>\n\n\n\n
qemu-img create -b<\/code> to create a new, temporary image to work in. This ensures that each build will run in a clean, fresh install, without any risk of contamination from previous builds. (The reason we did not do this for the initial step is that we want to save the bzr revisions pulled from Launchpad so we do not have to keep repeatedly pulling the old ones over for each new build. An alternative would be to keep the permanent shared repository on the host machine and export from that inside the virtual machine).<\/p>\n\n\n\n
The results from this can be seen on the MariaDB Buildbot pages<\/a>. Things are likely to to shuffle around as we extend
and refine this, but for now an example build can be seen here:<\/p>\n\n\n\nFull config<\/h2>\n\n\n\n
f_kvm_jaunty_x86= factory.BuildFactory()\nf_kvm_jaunty_x86.addStep(Compile(\n description=[\"making\", \"dist\"],\n descriptionDone=[\"make\", \"dist\"],\n logfiles={\"kernel\": \"kernel_2222.log\"},\n command=[\"sh\", \"-c\", \"\"\"\nkill -9 \"$(cat kvm_pid_2222)\"\n(exec sh -c \"echo \\$\\$ > 'kvm_pid_2222' ; exec kvm -m 2047 \\\n -hda \/kvm\/vms\/vm-jaunty-i386-makedist.qcow2 -redir 'tcp:2222::22' -boot c -smp 2 \\\n -cpu qemu32,-nx -nographic -net nic,model=virtio -net user\" \\\n <\/dev\/null >kernel_2222.log 2>&1) &\nsleep 15\nwhile : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done\nssh -p 2222 buildbot@127.0.0.1 'mkdir -p buildbot && cd buildbot && \\\n rm -Rf build && bzr co \"lp:~maria-captains\/maria\/mariadb-5.1-knielsen\" build && \\\n cd build && BUILD\/compile-dist && make dist && \\\n mv \"$(make show-dist-name).tar.gz\" ..'\n\"\"\"\n ],\n ))\nf_kvm_jaunty_x86.addStep(SetProperty(\n property=\"distname\",\n command=[\"ssh\", \"-p\", \"2222\", \"buildbot@127.0.0.1\",\n \"cd buildbot\/build && make show-dist-name\"],\n ))\nf_kvm_jaunty_x86.addStep(ShellCommand(\n description=[\"copying\", \"tarball\"],\n descriptionDone=[\"copying\", \"tarball\"],\n logfiles={\"kernel\": \"kernel_2222.log\"},\n command=[\"sh\", \"-c\", WithProperties(\"\"\"\nscp -P 2222 buildbot@127.0.0.1:buildbot\/%(distname)s.tar.gz .\nssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'\nwhile : ; do sleep 5; kill -0 \"$(cat kvm_pid_2222)\" || break; done\nrm -f kvm_pid_2222\n\"\"\")],\n ))\nf_kvm_jaunty_x86.addStep(Compile(\n description=[\"making\", \"bintar\"],\n descriptionDone=[\"make\", \"bintar\"],\n logfiles={\"kernel\": \"kernel_2222.log\"},\n command=[\"sh\", \"-c\", WithProperties(\"\"\"\nqemu-img create -b \/kvm\/vms\/vm-jaunty-i386-build.qcow2 -f qcow2 vm-tmp-2222.qcow2\nkill -9 \"$(cat kvm_pid_2222)\"\n(exec sh -c \"echo \\$\\$ > 'kvm_pid_2222' ; exec kvm -m 2047 -hda vm-tmp-2222.qcow2 \\\n -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio \\\n -net user\" <\/dev\/null >>kernel_2222.log 2>&1) &\nsleep 15\nwhile : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done\nssh -p 2222 buildbot@127.0.0.1 'rm -Rf buildbot && mkdir buildbot'\nscp -P 2222 %(distname)s.tar.gz buildbot@127.0.0.1:buildbot\/\nssh -p 2222 buildbot@127.0.0.1 'cd buildbot && tar zxf %(distname)s.tar.gz && \\\n cd %(distname)s && CC=\"gcc -static-libgcc\" CXX=\"gcc -static-libgcc\" \\\n CFLAGS=\"-O2 -fno-omit-frame-pointer -g\" CXXFLAGS=\"-O2 -fno-omit-frame-pointer -g\" \\\n .\/configure --prefix=\/usr\/local\/mysql --exec-prefix=\/usr\/local\/mysql \\\n --libexecdir=\/usr\/local\/mysql\/bin --localstatedir=\/usr\/local\/mysql\/data \\\n --with-server-suffix=1 --with-comment=\"(MariaDB - http:\/\/askmonty.org\/)\" \\\n --with-system-type=linux-gnu --enable-shared --enable-static --enable-thread-safe-client \\\n --enable-local-infile --with-big-tables --with-libwrap --with-ssl --without-docs \\\n --with-readline --with-extra-charsets=all --with-embedded-server --with-libevent \\\n --with-partition --with-zlib-dir=bundled --with-plugins=max-no-ndb && make -j3 && \\\n sudo rm -Rf \/usr\/local\/mysql && sudo make install && \\\n sudo mv \/usr\/local\/mysql \/usr\/local\/%(distname)s-Linux-x386 && \\\n tar zcf ..\/%(distname)s-Linux-x386.tar.gz -C \/usr\/local %(distname)s-Linux-x386\/'\nscp -P 2222 buildbot@127.0.0.1:buildbot\/%(distname)s-Linux-x386.tar.gz .\nssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'\nwhile : ; do sleep 5; kill -0 \"$(cat kvm_pid_2222)\" || break; done\nrm -f kvm_pid_2222\n\"\"\")],\n ))\nf_kvm_jaunty_x86.addStep(Test(\n description=[\"testing\", \"bintar\"],\n descriptionDone=[\"test\", \"bintar\"],\n logfiles={\"kernel\": \"kernel_2222.log\"},\n command=[\"sh\", \"-c\", WithProperties(\"\"\"\nqemu-img create -b \/kvm\/vms\/vm-jaunty-i386-install.qcow2 -f qcow2 vm-tmp-2222.qcow2\nkill -9 \"$(cat kvm_pid_2222)\"\n(exec sh -c \"echo \\$\\$ > 'kvm_pid_2222' ; exec kvm -m 2047 -hda vm-tmp-2222.qcow2 \\\n -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio \\\n -net user\" <\/dev\/null >>kernel_2222.log 2>&1) &\nsleep 15\nwhile : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done\nssh -p 2222 buildbot@127.0.0.1 'rm -Rf buildbot && mkdir buildbot'\nscp -P 2222 %(distname)s-Linux-x386.tar.gz buildbot@127.0.0.1:buildbot\/\nssh -p 2222 buildbot@127.0.0.1 'cd buildbot && \\\n sudo rm -Rf \/usr\/local\/mysql \/usr\/local\/%(distname)s-Linux-x386 && \\\n sudo tar zxf %(distname)s-Linux-x386.tar.gz -C \/usr\/local\/ && \\\n sudo ln -s %(distname)s-Linux-x386 \/usr\/local\/mysql && cd \/usr\/local\/mysql && \\\n sudo sudo chown -R mysql . && sudo chgrp -R mysql . && \\\n sudo bin\/mysql_install_db --user=mysql && sudo chown -R root . && \\\n sudo chown -R mysql data mysql-test && \\\n cd mysql-test && sudo su -s \/bin\/sh -c \"perl mysql-test-run.pl alias\" mysql'\nssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'\nwhile : ; do sleep 5; kill -0 \"$(cat kvm_pid_2222)\" || break; done\nrm -f kvm_pid_2222\n\"\"\")],\n ))\n\nbld_kvm_jaunty_x86 = {'name': 'kvm-jaunty-x86',\n 'slavename': 'knielsen-kvm-x86',\n 'builddir': 'kvm-jaunty-x86',\n 'factory': f_kvm_jaunty_x86,\n }\n\nc['builders'].append(bld_kvm_jaunty_x86)\n<\/pre>\n","protected":false},"excerpt":{"rendered":"