Next step: Python as an operating system. Like Chrome OS or Firefox OS, just with Python instead of a web engine. I'm a big enough Python fan that the idea really intrigues me.
(also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)
> (also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)
If you work anywhere even remotely related to Open Source, I'd recommend getting your employer to get a corporate subscription. It's one of the single most valuable news sources I read; it summarizes LKML and numerous other first-hand sources, as well as many conferences, summits, and other in-person events.
That would be interesting. This isn't quite freestanding Python, in that it's running on Grub so there is a lower layer, but it's still cool despite that.
Well, even Chrome OS and Firefox OS don't have their own kernels; they run on top of Linux.
If you were to build a Python OS, you'd probably start by writing PID 1 in Python. It would be a Python init, starting services written in Python, running gettys written in Python, and you'd log into a shell written in Python.
Hmm... I can actually see myself writing an init system in Python... (hell, I did exactly that at my last company.... I'd have to rewrite it from scratch, but that's no big deal...)
Every OS ever had a bootloader of some sort as far as I'm aware, in some cases you had to toggle them in through the front panel but I don't know of any OS that did not have a bootstrapping stage prepended.
There's the bootloader, there's the bios and there is the operating system. It's perfectly legal to do bios calls from a running operating system (though there will be a price to pay for sure). The bootloader is usually discarded after the boot. The bios doesn't 'run' in any way either until you call it, think of it as a library that you can call.
Really would have to do something about the GIL in that case unless you want to be restricted to one core - or run several independent interpreters NUMA-style.
Yeah, the GIL is the biggest reason we haven't tried to run any Python code on non-boot CPUs; we just run tiny C functions that return information to the boot CPU.
I really like Wunderlist. In the past few months, they seem to have fixed their syncing issues, so mobile and desktop are pretty seamless now. I keep tons of ideas in Wunderlist, and the search works pretty well if I just brain-dump some keywords into the todo item.
I also made a thing to suck all the Github repos you star into Wunderlist with their descriptions, since I always want to find something I starred in the past, and I've found Github's search lacking in this regard: https://github.com/ip2k/wunder-star
Next step: Python on bare metal ARM targets. Imagine loading Python interpreter to an uC, quickly hacking together some scripts and happily running some DIY project.
`sys` and especially `os` packages are out, maybe some high level features and something clever has to be done about interrupts, but that would be totally awesome
I seem to recall that the "Pi" in Raspberry Pi was originally "Py" and that the idea was that the system would boot straight into Python, similarly to how early personal computers booted into BASIC. This idea was later abandoned in favor of going with a full operating system.
The NodeMCU is an amazing piece of kit. I'd say it's the future for DIY "internet of things". I mean, a microcontroller with GPIO, spi, i2c etc. and built-in wifi hardware, running Lua, for $9. You can throw together "small screen showing live information from the internet" or "sensor uploading data to internet" for less than $20 total cost. The ecosystem is still very new, but it's growing really fast.
This could lower the barrier for js-only developers/designers/beginners to get into more low level programming, or to get them to play with some hardware a bit.
Yeah, we considered JavaScript for BITS for about three seconds, before dismissing the idea of doing systems programming in a language with no integers.
>Maybe one can run python on the network card rom too?
The network card ROM isn't microcontroller code, its just x86 code that runs on your main processor, called by the BIOS[1]. You could potentially hence have the OPs project run from the BootROM, if it would fit. It might be slightly tricky, as it runs as part of GRUB, which doesn't expect to find itself running from there.
Alternatively, the article mentions:
> Environments like Mirage OS (and other "just enough operating systems") could also add Python using the BITS code without too much difficulty, he said.
You could therefore probably add this code to iPXE [2] without too much trouble to get python on your BootROM, though I'm unsure how much of the C standard library you would need to borrow or reimplement. By the looks of it, iPXE doesn't have threads or local storage support, so you would be pretty limited in usability without a lot of work.
So maybe you can answer me this, how is memory management done in this environment? Are you free (heh) to do malloc and free or did you have to provide your own implementation? Is GRUB2 already in protected mode or there were some hacks needed on that area?
For memory management, GRUB2 fetches the memory map from the firmware (e820 under BIOS, BootServices::GetMemoryMap under EFI), and hands the usable memory regions to its memory allocator to allocate from. GRUB2's memory allocator then provides malloc, memalign, and free. (It also has some special functions for things like allocating low memory, needed for some system operations such as the CPU setup mentioned below.)
In the BIOS version, GRUB2 is entered in 16-bit real mode, and it transitions to 32-bit protected mode; it then has its own memory map. In the 32-bit and 64-bit EFI versions, the firmware transitions to 32-bit or 64-bit protected mode before calling GRUB2. (In all cases, there's no actual protection and the segment bases are all 0, though 64-bit mode does actually require a page table.)
However, GRUB2 normally only runs on the boot CPU ("BSP", bootstrap processor). We have to initialize the non-boot CPUs ("APs", application processors) ourselves. To do that, we allocate an aligned region of low memory, copy some bootstrap code and data into that memory, and send each processor an INIT (to reset them) and a startup inter-processor interrupt (SIPI) that forces them to jump to that address in low memory. We then transition them from 16-bit real mode to either 32-bit or 64-bit protected mode ourselves. (In the case of 64-bit protected mode, we set up paging by using the same page-table address that the firmware set up on the BSP, so that they all have the same memory map.) With that done, the AP can then run C code. Then we put the AP to sleep using mwait, waiting for work to do. When we want them to do something, we hand them a function pointer and a pointer-sized parameter.
This is EXACTLY what I needed right now for a project.
Does it already have a module to do the equivalent of an lspci? If not, where would I have to look to learn how to implement such a feature?
Note that PCI enumeration is a bit of an arcane art; if you want to know how to do it in practice on real systems with all their quirkiness, I'd suggest looking at how the Linux kernel enumerates PCI devices. (lspci in Linux userspace just walks the kernel-enumerated devices.) In particular, BITS doesn't have code to handle PCI-to-PCI bridges and enumerate the other side of them, or handle any quirks that crop up in real systems.
We have a very tiny Python pci module, which contains a brute-force bus-walk to find devices with a specific class; we're using that to find PCI-attached USB host controllers so that we can force them to "hand off" to the OS, which then causes the BIOS to stop polling them, which often causes the BIOS to stop doing frequent SMIs. (That allows us to tell if a BIOS's out-of-spec use of long-running SMI handlers is caused by USB handling.)
Simple PCI enumeration, though, is effectively just "walk every possible device, read the first four bytes of configuration space from function 0, if they're not 0xffffffff then they're vendor and product IDs, and then read the same from functions 1-7".
Thanks! That was definitely the most fun part of giving the talk; several people told me afterward that that was the point where the stuff I'd been talking about suddenly became real (accompanied by various degrees of "wat"), because it wasn't just talking to some abstract "hardware and firmware", it was drawing on the screen, and that made it real. And then following that up with Python-generated fractals really got people thinking about the old PEEK and POKE days.
My first programming environment was C under DOS, where you could create a pointer to video memory at 0xA0000000 or 0xC0000000 and start scribbling on the screen. I learned from a book ( http://smile.amazon.com/Microsoft-C-Programming-Robert-Lafor... ) which had an appendix showing how to draw the Mandelbrot set directly to video memory, so this was really nostalgic for me.
It is used for embedded systems but can work on a larger system too I suppose.
Basically it is just a kernel (for drivers) and on top (init) is Erlang VM.
The interesting aspect is an Erlang process is similar to a modern OS process -- has an isolated heap. So it kind of maps elegantly to an safe embedded multi-process OS.
Find the Erlang Factory talk by Frank Hunleth for more details.
Fortran predates the first examples of what we now think of as operating systems. After turning on the machine you fed in a deck of cards that comprised the Fortran compiler.
You know, how awesome would it be if a RPi booted up directly into Python in the same way old 8-bits booted up into BASIC? This could really help it better achieve the goal of being an educational instrument for young students.
I don't think BASIC being loaded on old systems had anything to do with education and was really just a convenience factor. I think qbrass makes some valid points in the sibling comment.
First thing I thought about - though if you're looking to do this with a Raspberry Pi as a purely educational tool, it may be easier just to create the pure-python tool as a layer on top of linux.
Depends on what you're trying to do; both are useful. Booting to Python under Linux gives you an exploratory environment with access to operating system services, but not necessarily one that lets you poke directly at hardware, unless you put Python in the kernel. (And even then, the kernel and drivers are doing things behind your back.) BITS gives you direct access to hardware and firmware, and it's intentionally single-tasking, running only what you tell it to.
So if you want a comfortable Python environment with more services and the ability to use the full power of Linux, you would want to boot to Linux and run Python (possibly with some extra modules); if you want raw access to hardware and firmware, but without any OS (with all the advantages and disadvantages that implies), you want BITS.
Holy cow. I've been looking for something like this. I'm currently trying to do board level testing on an embedded project with Python and the sysfs infrastructure in Linux. So far it's been decent but there are limitations to what I can do, such as testing all the ddr address lines.
Raw access to memory, MMIO, IO ports, registers, ACPI, and EFI should be pretty straightforward. If there's something you're trying to do that requires access to anything we don't expose in BITS, especially if it would require writing any new C code, please feel free to send us a patch, or just drop us a mail and we'll put it on the list to add.
This seems to be a trend - python, before it erlang-on-metal. Is there a stable higher-level language runtime that runs on bare metal (that I can boot in a VM)? Or, better yet, something that compiles arbitrary LLVM code into .ISOs for deployment?
These modern "GRUB + interpreter" attempts reminds me of CP/CMS. CP was a hypervisor and CMS was a small single-user operating system that runs on top of it (scriptable in REXX).
(also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)