Lotus Reader - The Hackernews Client

yekmer14 years ago

Our company uses HBase, Elastic Search, GitBlit, SmartFox Server, Jetty which have been by this bug, MySQL is said to be affected too http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

davidw yekmer14 years ago

Thank you for that link! I had been scratching my head about that server even though it wasn't mine to take care of (the other service I'm involved with here, that I helped plan, uses Postgres, which does not seem to have problems).

pilif14 years ago

I would love to see what's really causing this bug. We read so many times over the weekend to either reboot or just run that date command - but nobody is telling us what's causing the problem.

Also, seeing that other threaded applications had similar problems, I doubt this is a java issue - more likely a pthread, glibc or even kernel issue

gaiuspilif14 years ago

There is a good explanation here: http://serverfault.com/q/403732/58037

agwagaius14 years ago

That's predominantly about the kernel crash, not the high-CPU futex issue. One of the most maddening things about this is that there have been several different issues related to leap seconds on Linux, making it all the harder to get information.

altxwallypilif14 years ago

The patch that was shared on the lkml shows some insight on what is causing the issue. https://lkml.org/lkml/2012/7/1/27

Apparently the issues might be due "to the leapsecond being added without calling clock_was_set() to notify the hrtimer subsystem of the change", a possible fix being to patch kernel/time/timekeeping.c to be leapsecond aware.

ajayspilif14 years ago

This seems like the best explanation I've found so far: https://lkml.org/lkml/2012/7/1/203

pilifajays14 years ago

Agreed. Also it clearly accounts for the futex related load issues and it even gives nice and readable C code to see the problem happening.

This explains it for me. Thanks a lot for the pointer.

xxporpilif14 years ago

A good explanation from Reddit: http://www.reddit.com/r/programming/comments/vxmf7/time_arithmetic_you_win_again/c58lkmx

ecopoesis14 years ago

Hard to call this a Java bug when many other, non-Java things are affected. It's a critical Linux bug that causes futex to timeout, and anything that uses it to behave incorrectly.

https://lkml.org/lkml/2012/7/1/11

tommiecopoesis14 years ago

ecopoesis, you are not the only one saying that it's a linux bug instead of a java bug even though the link title says "Critical Linux bug that leads 100% CPU (leap second)".

Did the link title change from a Java title, like the article, to a Linux title to match the actual root cause?

davidwtommi14 years ago

> Did the link title change from a Java title, like the article, to a Linux title to match the actual root cause?

Yes, it did.

ww520ecopoesis14 years ago

It's probably that Java heavily utilizes the multi-thread support and the kernel bug is showing up as a Java bug. It just means Java really exercises the system's concurrent support.

geetee14 years ago

Hey, remember that time I spent a couple hours frantically checking logs and restarting services?

pjmlp14 years ago

This is a Linux kernel bug, not a JVM bug.

mcescalantepjmlp14 years ago

Yeah, NTP is Linux kernel, but the JVM is what's eating the CPU after the clock leap.

jbellismcescalante14 years ago

no, it's the kernel livelocking in response to a call made by the jvm

regularfry14 years ago

The easier to type 'sudo date -s "`date`" seemed to work for me.

j_col14 years ago

So that explains why the 12 cores on my Fedora workstation were maxed-out when I came to work this morning!

kzrdude14 years ago

So if the leap second was handled in userspace instead of the kernel, just like a normal ntp time update, all would have been fine. Why not just do that?

jhund14 years ago

I saw what is likely a related issue on one of our AWS EC2 instances, where exactly at midnight UTC there was a high percentage of 'steal' CPU time in our server monitoring charts.

I wonder if this was caused by another VM on the same physical box being hit by the bug and as a result stole CPU time from our VM.

I resolved the issue by moving to a different VM (Rebooting didn't help), to get away from my greedy neighbor.

More info here: http://blog.thinrhino.net.in/cpu-steal-time

e4014 years ago

On Sunday I noticed that Gerrit (code review, written in Java) was chewing through CPU on one of our servers. Just applied this it appears to have settled down.

coldskull14 years ago

well, our hadoop cluster went bonkers because of this bug....luckily it was on stage...not production!

danielhlockardcoldskull14 years ago

Yeah, I ended up rebooting our production hadoop cluster, it all came back up fine, and we don't have too many people using it yet.

freestyler14 years ago

There is a list of applications affected by this kernel bug http://blog.windfluechter.net/content/blog/2012/07/01/1481-100-cpu-load-due-leap-second

abc_lisper14 years ago

Does this happen on android too?

agentgt14 years ago

What a PITA

JVIDEL14 years ago

Oh man so that was causing it!

My rig crashed all weekend because of this POS bug, I had to boot back to Windows to get anything done (oh cmd, I really didn't miss you at all you insufferable bitch...)

Any fixes?

gcrJVIDEL14 years ago

There's a fix in the article.