JVM: Solving OutOfMemoryError with less Memory

At work we have 6 web applications (WAR) deployed in Glassfish v2.

In production we experienced sporadic java.lang.OutOfMemoryError: Java heap space under high load. We where sure that we did not have a classic Java memory leak since the used HEAP space decreased after some time and returned to normal. We suspected that the problem was related to our use of EHCache (which stores cached objects in HEAP space).

(Note: This blog post is a summary of two days of research – we tried many things and many numbers – on several more or less identical servers – so the numbers and values in this post is approximately correct – but you get the point)

The JVM, and therefor GlassFish, was running with max 768 MB of HEAP space and 385 MB of PermGEN space.

To reduce the possibility of getting OutOfMemoryError until we had worked out the issue, we decided to increase the maximum HEAP space the JVM could allocate (-Xmx).

GlassFish runs on a 4 core 32 bit server with Windows Server 2003 with 8 GB of RAM.

We increased the the max HEAP memory size to 1 GB (-Xmx=1024M). We started GlassFish with no applications deployed. Then deployed one application after the other – 6 WARs. All applications where deployed without problem and our Apps run fine. After some time the JVM suddenly died.

We found a crash JVM crashdump. We didn’t read it to carefully, but it talked about OutOfMemoryError. We did some more research and found out that it had died before the HEAP space had reached 1GB. We thought a solution was to set the initial HEAP size that the JVM should initialize. We told the JVM to initialize all the HEAP at startup (-Xms=1024M).

So now we had 1024MB HEAP and 385MB PermGEN which is a total of 1409MB.

When we then again started GlassFish (with no apps deployed), The JVM and GlassFish started up just fine. So we started to deploy applications – one by one. When GlassFish was in the middle of deploying the second application the JVM died.. So by allocating more memory up front, the JVM died with OutOfMemoryError earlier..

After a lot of research and reading this great post: http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html, this is how we concluded:

We took a closer look at the JVM crash dump:

java.lang.OutOfMemoryError: requested 884680 bytes for Chunk::new. Out of swap space?

It also says that the JVM crashed in this thread:

0x5be76800 JavaThread “CompilerThread1” daemon [_thread_in_native, id=6764, stack(0x5c1a0000,0x5c1f0000)]

We had configured the JVM to use a lot of memory for HEAP and PermGen. A Windows process can use max 2 GB total. The internals of the JVM (e.g its JIT compiler) needs its own memory, so do the DLLs loaded. Since so much of those 2GB was already allocated for HEAP/Permgen, windows said NO when the JVM asked for more memory inside CompilerThread1. When this happened, the JVM crashed with java.lang.OutOfMemoryError?: requested 884680 bytes for Chunk::new. Out of swap space?


Tell the JVM to use LESS memory..

maven deptools plugin 1.1 released

Version 1.1 of maven deptools plugin now supports maven 3 and the “maven enforcer plugin”

maven deptools plugin “…gives build error if maven resolves transient dependencies in such a way that the none-newest version is chosen.”

This plugin has turned out to be very useful in the company I work for.

The plugin can be found here: http://github.com/mbknor/deptools

maven deptools plugin – RC1 released

I’ve just released RC1 of a maven deptools plugin. (This is beta but I need real feedback)

“…Maven 2 plugin which gives build error if maven resolves transient dependencies in such a way that the none-newest version is chosen

At work we have all kinds of different dependencies problems related to transient dependencies…

More info here: http://wiki.github.com/mbknor/deptools/

Taking control over legacy code

The problem
Some years ago I faced a situation where a company’s main public webapplication ran on a legacy mainframe (OS/390) webserver. It was written in REXX.
The developers had to do the actual coding in a terminal window (3270).
If a developer wanted to code in a regular Text Editor (TextPad), he had to first download the sourcefile via FTP, edit it, then FTP the new version back up to the mainframe to test it. To compile the uploaded sourcefile he had to use a terminal window and navigate to the file (dataset), then disable and enable it to force a recompile of the file.
One other major problem with the FTP-solution was that different developers did overwrite each others changes when they uploaded their new files.
Since the source was not managed by any source control system, it was basically impossible to figure out who had change the code and why.
As you can see this was not an ideal situation.
The ideal solution
The ideal solution is for sure to rewrite the application from scratch with modern technology, but this was not an option for the Company. They felt that they had invested too much in the existing code and that it would take too long time to rewrite it. Not to mention that they would have been unable to create new stuff while porting the old stuff.
Taking control over the legacy code

Since it was not an option to rewrite the application we needed to make it as convenient as possible to work with it.

This is what I ended up doing:
We downloaded all the code and added it to SubVersion. Then we “defined” that that the version stored in SubVersion was the “master (correct) version” of the code, not the version stored on the mainframe.
Then I wrote a deployment tool in Java that automated the deployment-process.
Since we could not prevent other developers (in other teams) to directly edit the code on the mainframe we had to have a mechanism to prevent us from silently overwrite their code changes. This was a critical feature when selling the “idea” to my leader.
To detect this the deployment tool automatically added some metadata to the sourcefile when uploading it to the mainframe. This metadata contained a hash-value (crc, fingerprint) representing the exact state of the sourcecode when uploaded. This made it possible to validate the existing mainframe version of the file before overwriting it with new versions.
The metadata was generated inside a comment (/* metadata */) since the altered source file still needed to compile.
The deployment tool could also be used to compile the source remote on the mainframe. This was done by using a linux component called s3270 which lets you script the terminal session. Since we needed to run the deployment tool on windows, the deployment tool ran s3270 using cygwin.

Since the upload- and compileprocess was slow, we wanted to prevent us from uploading and compiling unchanged files.
To fix this we also included subversion url- and revision-info in the metadata. This made it possible to resolve which files had changed and only upload- and compile those.

By taking control over the source (adding it to SubVersion) and automating the deploy- and compile-process we ended up with an much better development environment.
This ended up only being the first step away from the mainframe. Today this old application still lives, only that it runs inside an Emulator running in Tomcat on a Windows Server.
I hope this blog post inspires someone.

Replacing Weblogic with Glassfish

At work we’re working on a rather big Java integration project. It consist of several applications, some exposing services over REST, others consuming REST services (JSF, Spring WebFlow).

In our development environment we run jetty and/or tomcat. Someone decided that we had to use Weblogic in Test- and Production-environment.
Here is a list of some of the problems Weblogic caused us:
  • It starts up/restarts extremely slow
  • The admin console is really slow
  • It takes forever to deploy to it
  • we had problems getting multiple datasources to different DB2-environments (OS390, AS400, Windows) to work at the same time.
  • Our applications ran slower than expected
  • We experienced strange problems related to ajax and richfaces which where impossible to trace down, since the problems where different on different Weblogic instances.
Today we managed to persuade our project leader that we should replace Weblogic with Glassfish v2.
Now everything is running much faster without problems on Glassfish in our Test environment. I really like what I have seen of Glassfish so far.


Tenkte jeg skulle tipse om opendns.com. Det er en rask, sikker og gratis dns-service.

Hva er DNS?
DNS står for domain name system og er protokollen datamaskinene bruker for å oversette fra domenenavn til IP-adresser.

Alle PC’er må vite om sin DNS server for å fungere på internett.

Det vanlige er å bruke den DNS serveren som man automatisk får tildelt fra ISP’en sin.

Hva er fordelen med opendns.com?

Den er ofte veldig mye taskere en ISP’ene sin DNS server noe som fører til at du får en raskere internett opplevelse.

Den er sikrere: de er veldig flinke til å tenkte sikkerhet og oppdatere serverene sine hvis sikkerhetshull oppdages. Dette beskytter deg for eksempel mot DNS cache poisoning.

De vedlikeholder lister over farlige nettsteder slik at de stopper deg hvis du er på vei mot et slikt farlig sted.

Hvordan bruke opendns.com?

Man legger inn okonfigurerer routeren sin til å benytte opendns.com sine dns-servere istedenfor de som man automatisk har fått tildelt fra ISP’en. Da vil alle maskiner som kobbler seg på routere din automatisk benytte opendns.com sine DNS servere. Man endre konfigen i routeren en gang og trenger aldri mer å tenke på det…

Joel Spolskys mening om project managers

Jeg hørte på podcasten StackOverflow for første gang i dag. Jeg likte den godt og kommer helt sikkert til å fortsette å høre på den i fremtiden.

Jeg likte spessielt godt det Joel Spolsky sa om hans mening om project managers.
Hele episoden er på over en time, så jeg tok meg friheten til å klippe ut de aktelle 5m35s.

Hvis teknologien er på min side skal dere kunne høre på det her

Google Calendar for mobil på Norsk

Jeg har begynt å bruke Google Calendar og liker det..

Jeg bruker hovedsaklig web-grensesnittet på pc’en, men da jeg skulle sjekke ut Google Calendar fra mobilen min (Sony Ericsson W910i) oppdaget jeg et problem: Det var umulig å legge inn nye hendelser.

Jeg kunne se eksisterende hendelser, men ikke legge inn nye. Dette var da merkelig.. I all dokumentasjon stod det at man skulle kunne legge til hendelser også.

Løsningen var å endre språket på Google Calendar fra Norsk til Engelsk. Dette må gjøres fra PC.

Når språket er endret til Engelsk, dukker “Add event”-knappen opp på mobilen.

PS: Husk å fikse datoformatet til “31/12-2008” når du endrer språk til Engelsk..

Det var imponerende lett å legge inn hendelser fra mobilen.

Skrev inn “Kaffe med venner 29/10 13:00” som tekst, klikket “Add event”. Det var alt som skulle til for å få lagt inn hendelsen. Funket som F..

Hvordan fjerne Spyware?

(Les hele teksten før du gjør noe som helst)

Bakgrunn for blogposting

Jeg fikk for litt siden i oppdrag av en nabo å fikse laptop’en deres. Den var infisert av spyware. Jeg fikset problemet og laptop’en ble “så god som ny”. Siden det er andre (nære og fjerne, inklusiv naboens søster) som kan ha nytte av en forklaring på hvordan jeg fjernet spywaren, tenkte jeg at jeg kunne skrive det ned her.

Jeg kommer ikke til å gå i dybden her, men skal prøve kort og konsist å forklare.

Hva er Spyware?

Spyware, også kalt trojaner, er dataprogrammer som blir installert på din pc uten at du vill det. Deres oppgave er å spionere på deg eller å vise reklame du ikke vil ha.

Hvordan får man spyware?

Man kjører spyware programmet på pc’en uten at man vet at det er det man gjør. Eksempel:

  • Noen sender deg en epost til et “Elektronisk postkort”. Du klikker på det og får se postkortet, men uten at du merket noe ble et spyware-program installert på pc’en din.
  • Du går inn på en nettside som uten at du merker det utnytter et “hull” i nettleseren din (Internet Explorer, Firefox, osv) og kjører programmet på pc’en din uten at du merker det.

Har jeg spyware?

Det er ikke alltid så lett å vite, men hvis ting går veldig tregt så er det sansynlig at noe er galt. Hvis det kommer frem vinduer med reklame fra tid til annen uten at du gjør noe for at det skal skje, så har du helt sikkert spyware.

Hvordan fjerner jeg spyware?

Først la meg si at enkelte typer spyware er vanskelig eller umulig å bli kvitt. Men her skal jeg forklare hvordan jeg fikset naboens laptop.

Jeg lastet ned tre programmet, en antivirus og to programmer som leter etter og fjerner spyware.

AVG Anti-Virus Free Edition

AVG er et helt gratis antivirus program som fungerer ypperlig. Siden det er gratis, vil den automatiske oppdateringen aldri slutte å virke siden abonementet ditt aldri går ut..

AVG Anti-Virus Free Edition kan lastes ned her:


  • Du må ALDRI ha mer en ett antivirus program installert på maskinen din.
  • Et antiviprogram som ikke blir gjevnlig oppdatert er ubrukelig. Forklaring: Det kommer nye virus hver dag og antivirusprogrammet kan bare finne de virusene som den vet om.
  • Hvis du skal installere ny antivirus må du først avinstallere det gamle antivirusprogrammet før du installere det nye.

Slik bruker du det:

  • (Avinstaller eventuel annen antivirusprogram)
  • installer programmet
  • Sørg for at det er oppdatert – klikk på Update
  • scan hele maskinen så den leiter etter eventuell viruser

Spybot – Search & Destroy

Spybot er et gratis antispywareprogram, dvs at det kan lete etter kjente spywareprogrammer of fjerne det den finner.

Spybot – Search & Destroy kan lastes ned her:

Spybot er litt knotete å bruke men prøving og feiling gjør mester 🙂

Slik bruker du det:

  • Last ned og installer
  • Start programmet
  • oppdater (for å hente ned oppdatert info om nye spyware-programmer) , kryss av for alt når du skal opdatere.
  • Scan
  • Merk alt den finner.. Ikke alt den finner er kritisk… og fjern det.

Ad-Aware 2008 Free

Ad-Aware 2008 Free er et annet gratis antispywareprogram som typisk finner litt andre spyware-problemer en Spybot.

Ad-Aware 2008 Free kan lastes ned her:

Dette programmet er litt enklere å bruke en spybot.

Gjør det samme her som med spybot: installere, oppdater, scan..


Hvis du allerede har et ok og oppdatert antivirusprogram kan du oppdatere dette og scanne hele pc’en. Hvis ikke, avinstaller det og installer AVG isteden for. Oppdater så AVG og scan hele pc’en.

Når du er ferdig med å scanne etter virus, starter du Spybot, oppdaterer dette og scanner etter spyware. fjern alt den finner.

Kjør så Ad-aware, oppdater og scan. fjern alt den finner.

Start maskinen på nytt og kjør AVG, Spybot og Ad-aware enda en gang..

Når de (nesten) ikke finner noe mer, så er pc’en din mest sansynlig frisk og fri for spyware.

lykke til


Jeg innser jo at teksten over er langt fra forklarende nok, men kanskje den kan være til hjelp.. Jeg håper det..