Every now and then you might need a Linux distribution for a CPU that you do not have. As an example you might want to run an ARM distro on an x86_64 machine. This is not only possible but quite simple. Here's how you would do it on Ubuntu or Debian.
First install the qemu-user-static package and create a subdirectory to hold your distro install. Then run the first stage install. Using Debian sid as an example:
sudo debootstrap --arch=armhf --foreign sid sid-arm
Once that has finished, copy the qemu binary inside this install. This makes it possible to run ARM binaries transparently on an Intel CPU.
sudo cp /usr/bin/qemu-arm-static sid-arm/usr/bin
Then finish the bootstrap operation.
sudo chroot sid-arm /debootstrap/debootstrap --second-stage
And now we are done. Chrooting in sid-arm and running uname -a prints this:
Linux userland 4.1.0-2-generic #2-Ubuntu SMP Wed Jul 22 18:19:08 UTC 2015 armv7l GNU/Linux
At this point you can use the chroot as if it were a native installation. You can install packages with apt, compile stuff with gcc and do anything else you might prefer. In theory you could even run it as a container with systemd-nspawn, but unfortunately qemu is missing some functionality to do that. The bug report is here. The only downside of this setup is that because the CPU is emulated, all binaries run a bit slow.
Saturday, July 25, 2015
Tuesday, July 21, 2015
List of achievements for an open source project
- [ ] first commit
- [ ] making the code public
- [ ] first checkout by someone else
- [ ] first user
- [ ] first bug report
- [ ] first mention in a blog post by someone you don't know
- [ ] first submitted patch
- [ ] approval to Debian/Fedora
- [ ] a magazine article mentioning your project
- [ ] a magazine article about your project
- [ ] first conference presentation
- [ ] a project subreddit is created
- [ ] a conference presentation about your project held by someone you don't know
- [ ] a bug report/patch from Linus Torvalds/someone else well known
- [ ] RMS claims your project is anti-freedom
- [ ] the project gets forked
- [ ] an O'Reilly animal book gets published
- [ ] a conference wholly about your project
- [ ] someone reimplements your project in a new language in 1/10th of the time it took to write the original one
- [ ] making the code public
- [ ] first checkout by someone else
- [ ] first user
- [ ] first bug report
- [ ] first mention in a blog post by someone you don't know
- [ ] first submitted patch
- [ ] approval to Debian/Fedora
- [ ] a magazine article mentioning your project
- [ ] a magazine article about your project
- [ ] first conference presentation
- [ ] a project subreddit is created
- [ ] a conference presentation about your project held by someone you don't know
- [ ] a bug report/patch from Linus Torvalds/someone else well known
- [ ] RMS claims your project is anti-freedom
- [ ] the project gets forked
- [ ] an O'Reilly animal book gets published
- [ ] a conference wholly about your project
- [ ] someone reimplements your project in a new language in 1/10th of the time it took to write the original one
Wednesday, July 15, 2015
Strange requirements of low level packages
As some of you might know, I'm the main developer of the Meson build system. It is implemented in Python 3 but has strict requirements on dependencies. Every now and then this confuses people so here's an explanation why we do things slightly unconventionally.
One of the greatest things about modern development environments is the number of third party packages and how easy they are to use. This is why Meson has spent a lot of effort in creating a simple way to get dependencies automatically on multiple platforms. This system is called Wrap, but this text is not about that. Instead it's about the lack of dependencies in Meson itself. The policy is that in order to run Meson and its test suite, you are only allowed to depend on Python standard library. No external dependencies are allowed. For non-core stuff (coverage, static analysis) external dependencies are ok.
This seems a bit strange and has caused us to have to reinvent some wheels. There are valid reasons for this, though, which have to do with the fact that packages at the lowest levels of the software stack have requirements that regular applications don't. The most important of these is that a build system needs to work in very limited distros and environments.
Let's look at Mer project for an example. It aims to provide an extremely slimmed down Linux distribution for use in mobile applications. They support only a small number of packages by design. There are only 8 Python packages in total. Six are extension packages and those are not useful for a build system. If we wanted to get Meson there, what would it actually mean?
If Meson depends on any other package, it can not be accepted into Mer. Since source bundling is prohibited and the system does not allow net access during package creation, it just can't be done. Either you disable the part that requires the extension or you don't get your package accepted. You might get an exception but you need to apply for it and it takes time an effort for many people. If only the unit test system depends on external packages you can disable it but then you can't tell if your project is working correctly. This is not acceptable either.
There are a bunch of distros like this. Then there are the BSDs that might not have your dependencies in their repos. Rather than force them all to deal with this dependency issue we instead write our code so that it has no external dependencies. This means more work for us but less for everyone else. Luckily Python 3's standard library is very good and has almost everything we need.
Another use case that many people don't think about is that a lot of development is done on machines that neither have Internet access nor the capability to install additional software. Several corporations have requirements such as these for real or imagined security reasons. Even some OSS development is done like this. Requesting permission to use a new dependency can take six weeks.
I'm not saying that this is right or wrong or even sane. What I'm saying is that this is reality and that you ignore reality at your own peril.
In an environment like this getting Meson approved for use takes only one round of bureaucracy. Every dependency requires an additional round (if you are lucky). To reduce friction and ease uptake, we must again cut down on our dependencies. The easier Meson is to deploy, the more people are willing to switch to it.
There are other advantages, too. It is easier to get contributions from Windows and OSX developers since they can hack on the code without any additional steps. You'd be surprised how tough it can be to download Python dependencies, even with setuptools. Deployment is easier, too: "Download Python 3 and Meson, run" as opposed to "download Python 3 and Meson, use setuptool to move it places, then install these dependencies and then set up these paths and ...".
Because of these reasons we don't use external dependencies. This is the price you pay for being at the very bottom of the software stack.
One of the greatest things about modern development environments is the number of third party packages and how easy they are to use. This is why Meson has spent a lot of effort in creating a simple way to get dependencies automatically on multiple platforms. This system is called Wrap, but this text is not about that. Instead it's about the lack of dependencies in Meson itself. The policy is that in order to run Meson and its test suite, you are only allowed to depend on Python standard library. No external dependencies are allowed. For non-core stuff (coverage, static analysis) external dependencies are ok.
This seems a bit strange and has caused us to have to reinvent some wheels. There are valid reasons for this, though, which have to do with the fact that packages at the lowest levels of the software stack have requirements that regular applications don't. The most important of these is that a build system needs to work in very limited distros and environments.
Let's look at Mer project for an example. It aims to provide an extremely slimmed down Linux distribution for use in mobile applications. They support only a small number of packages by design. There are only 8 Python packages in total. Six are extension packages and those are not useful for a build system. If we wanted to get Meson there, what would it actually mean?
If Meson depends on any other package, it can not be accepted into Mer. Since source bundling is prohibited and the system does not allow net access during package creation, it just can't be done. Either you disable the part that requires the extension or you don't get your package accepted. You might get an exception but you need to apply for it and it takes time an effort for many people. If only the unit test system depends on external packages you can disable it but then you can't tell if your project is working correctly. This is not acceptable either.
There are a bunch of distros like this. Then there are the BSDs that might not have your dependencies in their repos. Rather than force them all to deal with this dependency issue we instead write our code so that it has no external dependencies. This means more work for us but less for everyone else. Luckily Python 3's standard library is very good and has almost everything we need.
Another use case that many people don't think about is that a lot of development is done on machines that neither have Internet access nor the capability to install additional software. Several corporations have requirements such as these for real or imagined security reasons. Even some OSS development is done like this. Requesting permission to use a new dependency can take six weeks.
I'm not saying that this is right or wrong or even sane. What I'm saying is that this is reality and that you ignore reality at your own peril.
In an environment like this getting Meson approved for use takes only one round of bureaucracy. Every dependency requires an additional round (if you are lucky). To reduce friction and ease uptake, we must again cut down on our dependencies. The easier Meson is to deploy, the more people are willing to switch to it.
There are other advantages, too. It is easier to get contributions from Windows and OSX developers since they can hack on the code without any additional steps. You'd be surprised how tough it can be to download Python dependencies, even with setuptools. Deployment is easier, too: "Download Python 3 and Meson, run" as opposed to "download Python 3 and Meson, use setuptool to move it places, then install these dependencies and then set up these paths and ...".
Because of these reasons we don't use external dependencies. This is the price you pay for being at the very bottom of the software stack.
Tuesday, July 14, 2015
The fable of mashed potatoes
Once upon a time in a faraway land in a kingdom much like ours lived a carpenter. Every day he worked for a small company with his fellow carpenters. This company produced book shelves. The recent increase in population literacy rates had made books very popular so there was a growing need for bookshelves.
The bookshelves they manufactured were quite popular, and their customers really seemed to appreciate their quality. Obviously making bookshelves was not rocket science, mostly because rocketry had not been invented yet, but also because it was all in all a fairly well understood problem. The company's success made it possible for them to hire a few more carpenters. And then a few more.
Eventually someone in the management chain started wondering about how to make things better. Faster. More organised. So he went looking for a bookshelf expert to lead this voyage towards unknown bookshelf frontiers. Eventually such a person was found, hired and initiated. Soon thereafter he gathered all of the carpenters of the company into one room, for he had an announcement to make.
The bookshelf expert gathered everyone around him in a tight circle. He was giddy and excited to spring forth his major plan. All the carpenters were excited as well, there was electricity in the air. Then the expert finally started talking. First he talked about the past, then the present and finally came time for the future of all of storage space.
- From now on, we shall make all of our bookshelves ...
Every carpenter in the room leaned forward. They held their breath.
- ... out of mashed potatoes!
They kept holding held their breath, but for a different reason.
No-one can really remember how the rest of the talk went. One imagines the expert talked about the glorious future this would bring about. Then he congratulated everyone and left, presumably for a meeting with other mashed potatoes furniture specialists.
Our friend the carpenter felt puzzled. He wondered about the workshop for a while and then talked to his boss. He was frankly completely baffled. You couldn't even make a pencil stand out of mashed potatoes, let alone a bookshelf.
- He's a world class expert on potatoes, the boss said, he knows what is best so we should be doing what he says.
- What? This has nothing to do with potatoes at all. There's no way to make a bookshelf out of mashed potatoes, for starters it won't hold any weight, and also ...
The boss waved his hand at the carpenter.
- Have you ever tried doing a bookshelf out of mashed potatoes? I mean actually, really in practice?
- No, but, the carpenter tried to continue before being interrupted
- Well that's a very negative attitude to be having. Why are you being so negative? It affects poorly on your co-workers. Look around you, they are all working on their next projects and getting shelves done.
The carpenter tried to think of something to say, but the sheer absurdity of the situation got the better of him.
- Well back to work, we have products to ship, the manager said and left the workshop.
The sun went down and came back up the following day. Our carpenter had decided to make the best of this situation. Surely, he thought, someone would notice the flaw inherent in a mashed potato based storage unit. He didn't have long to wait. Soon the expert tapped him on the shoulder and asked him to come to the next room.
- I have been told, he said in a slightly judgmental tone, that you have raised some issues with my new plan.
The carpenter tried to follow up but was cut short.
- Apparently you have been telling people that my potatoes have mildew.
- Err, what? I have said nothing of the kind.
- I will have you know that all of these potatoes are the finest in the world. I have grown them myself with my own two hands. I have won several awards for potato farming.
- I'm sure you have, but ...
The expert cut him off again. He would not have any dissenting voices here. He went on explaining in great length how potatoes are the best nutritional source, how much time and effort he had put in each single potato and a blurry of other potato related facts.
- Now I'm going to let this infraction against potatoes go unnoticed, but do remember that a negative attitude such as yours will cause problems sooner or later. I highly recommend you fix it. Now, back to work.
And to work he went, among with all the rest of the team. They started producing mashed potato bookshelfs as planned and even scrapped already made but not yet delivered wooden shelves for new potatoe'd ones. And they were terrible. Upon seeing them their customers immediately demanded their money back. Some shelves did make it into customers apartments where they did not hold books but mostly just stank and attracted flies. Which caused the customers to demand their money back with interest.
As all sources of income dried up and management refusing to go back to wood based shelves (because "we have spent so much on this technology" and because "going back now would make us look weak"), it did not take long for the company to shut down. When the workshop was being cleaned for the last time, the carpenter's boss spoke with a sad timbre in his voice.
- Everything was going so well and now we have nothing. How can this be?
The carpenter had kept his opinions under his hat since the confrontation but as there was nothing left to lose, he figured he'd make one last appeal to sanity.
- Because making bookshelves out of mashed potatoes is a massively stupid idea that can not possibly work. Even a child can see that immediately. Remember how I tried to tell you this when things started?
The boss straightened his back and squinted. For a split second you could see tiny sparks shooting from his eyes.
- That's exactly it! You never gave our new strategy a chance! You had it doomed before it even started and then worked actively against it! Your negative attitude is the reason we are now all out of a job. You destroyed the livelihood of dozens of people! Are you proud of yourself?
The carpenter was not. He had failed. Miserably.
Failed to explain to a grown man that making a bookshelf out of mashed potatoes was a terrible idea.
The bookshelves they manufactured were quite popular, and their customers really seemed to appreciate their quality. Obviously making bookshelves was not rocket science, mostly because rocketry had not been invented yet, but also because it was all in all a fairly well understood problem. The company's success made it possible for them to hire a few more carpenters. And then a few more.
Eventually someone in the management chain started wondering about how to make things better. Faster. More organised. So he went looking for a bookshelf expert to lead this voyage towards unknown bookshelf frontiers. Eventually such a person was found, hired and initiated. Soon thereafter he gathered all of the carpenters of the company into one room, for he had an announcement to make.
The bookshelf expert gathered everyone around him in a tight circle. He was giddy and excited to spring forth his major plan. All the carpenters were excited as well, there was electricity in the air. Then the expert finally started talking. First he talked about the past, then the present and finally came time for the future of all of storage space.
- From now on, we shall make all of our bookshelves ...
Every carpenter in the room leaned forward. They held their breath.
- ... out of mashed potatoes!
They kept holding held their breath, but for a different reason.
No-one can really remember how the rest of the talk went. One imagines the expert talked about the glorious future this would bring about. Then he congratulated everyone and left, presumably for a meeting with other mashed potatoes furniture specialists.
Our friend the carpenter felt puzzled. He wondered about the workshop for a while and then talked to his boss. He was frankly completely baffled. You couldn't even make a pencil stand out of mashed potatoes, let alone a bookshelf.
- He's a world class expert on potatoes, the boss said, he knows what is best so we should be doing what he says.
- What? This has nothing to do with potatoes at all. There's no way to make a bookshelf out of mashed potatoes, for starters it won't hold any weight, and also ...
The boss waved his hand at the carpenter.
- Have you ever tried doing a bookshelf out of mashed potatoes? I mean actually, really in practice?
- No, but, the carpenter tried to continue before being interrupted
- Well that's a very negative attitude to be having. Why are you being so negative? It affects poorly on your co-workers. Look around you, they are all working on their next projects and getting shelves done.
The carpenter tried to think of something to say, but the sheer absurdity of the situation got the better of him.
- Well back to work, we have products to ship, the manager said and left the workshop.
The sun went down and came back up the following day. Our carpenter had decided to make the best of this situation. Surely, he thought, someone would notice the flaw inherent in a mashed potato based storage unit. He didn't have long to wait. Soon the expert tapped him on the shoulder and asked him to come to the next room.
- I have been told, he said in a slightly judgmental tone, that you have raised some issues with my new plan.
The carpenter tried to follow up but was cut short.
- Apparently you have been telling people that my potatoes have mildew.
- Err, what? I have said nothing of the kind.
- I will have you know that all of these potatoes are the finest in the world. I have grown them myself with my own two hands. I have won several awards for potato farming.
- I'm sure you have, but ...
The expert cut him off again. He would not have any dissenting voices here. He went on explaining in great length how potatoes are the best nutritional source, how much time and effort he had put in each single potato and a blurry of other potato related facts.
- Now I'm going to let this infraction against potatoes go unnoticed, but do remember that a negative attitude such as yours will cause problems sooner or later. I highly recommend you fix it. Now, back to work.
And to work he went, among with all the rest of the team. They started producing mashed potato bookshelfs as planned and even scrapped already made but not yet delivered wooden shelves for new potatoe'd ones. And they were terrible. Upon seeing them their customers immediately demanded their money back. Some shelves did make it into customers apartments where they did not hold books but mostly just stank and attracted flies. Which caused the customers to demand their money back with interest.
As all sources of income dried up and management refusing to go back to wood based shelves (because "we have spent so much on this technology" and because "going back now would make us look weak"), it did not take long for the company to shut down. When the workshop was being cleaned for the last time, the carpenter's boss spoke with a sad timbre in his voice.
- Everything was going so well and now we have nothing. How can this be?
The carpenter had kept his opinions under his hat since the confrontation but as there was nothing left to lose, he figured he'd make one last appeal to sanity.
- Because making bookshelves out of mashed potatoes is a massively stupid idea that can not possibly work. Even a child can see that immediately. Remember how I tried to tell you this when things started?
The boss straightened his back and squinted. For a split second you could see tiny sparks shooting from his eyes.
- That's exactly it! You never gave our new strategy a chance! You had it doomed before it even started and then worked actively against it! Your negative attitude is the reason we are now all out of a job. You destroyed the livelihood of dozens of people! Are you proud of yourself?
The carpenter was not. He had failed. Miserably.
Failed to explain to a grown man that making a bookshelf out of mashed potatoes was a terrible idea.
Friday, July 3, 2015
How many is too many?
Trivia question: when you run a random configure scripts, how many processes does it spawn?
Before reading further, try to think of the answer by yourself. Just do it. Pick a number you think is the correct answer.
Measuring this is simple. You can do it with the following command:
strace -c -f -e trace=fork,vfork,clone,execve bash -c ./configure
Different configure scripts have different results, of course. I ran this on the configure script of RPM, which I happened to have handy.
And the correct answer is ...
... 6611.
Aren't you glad that processes are cheap?
Before reading further, try to think of the answer by yourself. Just do it. Pick a number you think is the correct answer.
Measuring this is simple. You can do it with the following command:
strace -c -f -e trace=fork,vfork,clone,execve bash -c ./configure
Different configure scripts have different results, of course. I ran this on the configure script of RPM, which I happened to have handy.
And the correct answer is ...
... 6611.
Aren't you glad that processes are cheap?
Subscribe to:
Posts (Atom)