Take off your BOV, and put back the stock recirc valve. The STi's ECU settings are calibrated precisely for the amount of air which the stock recirc valve will plumb back into the system. So if you bypass this with a BOV which releases this amount of air instead of recirculating it, you will temporarily run extremely rich, thus causing your rpm fluctuations. You will need engine management (e.g. ECU reflash, piggyback chip, etc.) if you insist on keeping the BOV.
Anyway, the stock recirc valve can take a lot more pressure than the stock STi turbo can achieve. Installing an aftermarket BOV will only hurt performance on the STi, unless you have changed the turbo and you are doing a lot more than 1.6 bar.
I suggest you check out the Subaru Club Malaysia forums. Those guys are very helpful and you will find a wealth of technical information and advice.
http://forums.mysubaruclub.com/