Page #1:Why Google Said Linux is Slow
Google found Linux is slow!
Hello traffic, here i have started a series of Pages where I will be writing every weekend one random but interesting page for you guys to read.
Enjoy reading…
Here’s a problem nobody has except google, linux is taking a minute to restart you might say why? my linux sticks seconds to restart so those guys have 16+ drives nvme ssd drives on their linux systems.
When linux sends a signal to that drive to shut down and whatever it has in its cache that takes around 4.5 to 5 seconds, wow i was surprised it was like just five seconds to shut out drive so when you have 16+ drives that just adds up to a minute right because linux is doing this synchronously and it’s fascinating to me when i came to know about this how google is trying to optimize the heck of every single piece of the software stack.
About this synchronicity in linux google has a problem with linux server reboots too slow due to too many nvme drives hyperscale problems these days linux servers taking too long to reboot due to having too many nvme intros thankfully google is working on an improvement to address this where some of their many driver server can take more than one minute for linux to kernel to shut down so nobody cares.
we’re starting service like if i’m starting a server you know how often your start actual server you keep them running for months and if you actually want to restore them yeah that’ll take a minute or two but google cannot afford that right especially if you want to have availability and high availability all the time you want these machines to come back as fast as possible one minute can cost you a lot of money if you’re google so google engine are proposing an asynchronous shutdown interface for the next corner currently the linux kernel shutdown api and the bus level are synchronous but which can cause problems like google reports with having too many nvme storage drives in a single server due to synchronous nature during the shutdown handling each nvme can take about 4.5 seconds to shut down so add that up and and you’re going to get around a minute so this brings us back to just the nature of programming when you have synchronous apis.
Let’s assume the linux operating system is doing a loop it’s like i have 16 devices let’s loop through each one of them and send this shutdown signal right and you have to wait for the sync for the signal to come back for the api to respond to you that okay the shutdown has happened right and that takes five seconds then only you can move out of your loop and go to the next item effectively that’s how synchronous apis work, you send and it’s blocking you so you cannot move to the next uh thing unless you actually wait for it so 4.5 second and then 4.5 seconds again 4.5 again 4.5 second again 4.5 second then a minute later you’re just stuck so changing that to asynchronous will just loop all of them send a signal to all of them concurrently almost concurrently not quite but at the same time send a signal to all of them and all these requests just sent in the loop and then you send all these requests you don’t wait for everyone and you get to back come sort of i call back whatever the implementation is right but then eventually once you have all of these jobs you’re gonna get back the events but the devices will shut down locally you know in parallel so the entire fleet of devices will shut down in around 4.5 seconds or less be generous maybe six or seven you know take take into consideration the actual time to loop and send the signal and get the callback logic and all that so yeah the whole thing just you can save an entire minute obviously there’s there must be logic to handle that at the api level you know it’s like to wait for all of these devices or you cannot just send this and go to shutdown you have to actually wait you still need to wait for all these devices to actually shut down but the weight here is now much quicker because now you sent these requests and then you’re now waiting for all of them to reply to you.
You’re not just sending one request and you’re waiting and then sending and waiting and sending and waiting no you just sent everything and then you wait so yeah the fix looks simple enough i think it’s safe i just don’t know practically speaking if you have like a thousands of nvme drives or a lot of these drives in the one server i don’t know if we’ll be still have the same effect can you send thousand concurrent shutdown signals to in parallel almost to all those devices and just wait and then how does this the linux manage that there is all this asynchronous thing on linux and yeah 16 devices is not really that bad it’s just that’s that’s where you really need to you know to balance synchronous workload versus asynchronous, yeah asynchronous is great you just send all these requests and i don’t care i can do some other things but if you send too many requests then you really need to come back and manage those request and then it depends on the in the asynchronous program itself and linux might have a great one that doesn’t have any side effects but i can’t imagine that sending like 10,000 requests at the same time asynchronously it would be a good idea right at the end of the day you need to mix synchronous which is the synchronous semantic is much simpler definitely memory efficient i would assume you know because of this is increased you still need some sort of a structure to manage the out that the result that come back to you.
Guys what do you think about this i found this interesting i love these kind of stuff so thought of sharing it with you all.
Here is the Google’s patch.
The above link might give you a warning once you click on but feel safe to move forward, this link does not contains any kind of malicious thing.
