Thread: thoughts on adding ConcurrentMap methods to StoredMap?


Permlink Replies: 12 - Pages: 1 - Last Post: Mar 29, 2007 11:45 AM Last Post By: jahlborn
jahlborn

Posts: 34
Registered: 02/12/07
thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 12, 2007 8:31 AM
Click to report abuse...   Click to reply to this thread Reply
as far as i can tell, StoredMap has the potential to implement the extra methods in the jdk 1.5 ConcurrentMap interface. Database already has a putNoOverwrite method which would be the implementation of putIfAbsent. Personally, however, i am more interested in the other methods, all of which basically do their work only if the current value matches a given old value. this enables safe concurrent updates to a StoredMap even outside of transactions, basically using an optimistic scenario:

- read existing key,value (hold no locks)
- copy current value and modify the copy
- write modified value only if current value matches what is now in the database (this line within the context of a RMW lock)

any idea if this could be incorporated in the je codebase anytime soon? i may end up implementing it in our codebase, although the je api is not easy to extend as most classes/useful methods are package protected (arg!).
greybird

Posts: 1,296
Registered: 07/13/06
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 12, 2007 3:43 PM   in response to: jahlborn in response to: jahlborn
Click to report abuse...   Click to reply to this thread Reply
as far as i can tell, StoredMap has the potential to
implement the extra methods in the jdk 1.5
ConcurrentMap interface.

Yes, implementing ConcurrentMap would be really nice, and not very difficult. The main problem we have right now is that we do not depend on Java 1.5, so we could not actually implement the 1.5 ConcurrentMap interface.

We could add the methods, however, without implementing the interface. Are you more interested in the methods (the functionality) or the implementation of the standard Java ConcurrentMap interface?

If it is important to implement the standard Java ConcurrentMap interface, then perhaps we could implement it as separate Java 1.5-only classes -- StoredConcurrentMap and StoredConcurrentSortedMap. This would extend StoredMap and StoredSortedMap, and would only be usable from Java 1.5 and later.

Database already has a
putNoOverwrite method which would be the
implementation of putIfAbsent.

Yes, this would be easy.

Personally, however,
i am more interested in the other methods, all of
which basically do their work only if the current
value matches a given old value. this enables safe
concurrent updates to a StoredMap even outside of
transactions, basically using an optimistic
scenario:

- read existing key,value (hold no locks)
- copy current value and modify the copy
- write modified value only if current value matches
what is now in the database (this line within the
context of a RMW lock)


Since we need to write-lock (RMW lock) the record before writing it, I don't see any advantage to reading the record without locking initially. We might as well just:

- read with RMW
- if the current value equals the given value, then write

This implementation requires the use of a cursor or a transaction.

any idea if this could be incorporated in the je
codebase anytime soon?

We don't have any plans to do this, but I'm happy to help you if I can understand better exactly what your needs are.

i may end up implementing it
in our codebase, although the je api is not easy to
extend as most classes/useful methods are package
protected (arg!).

Yes, those classes were not designed to be extended, although you can always do so by modifying the source code.

Mark
jahlborn

Posts: 34
Registered: 02/12/07
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 9:34 AM   in response to: greybird in response to: greybird
Click to report abuse...   Click to reply to this thread Reply
as far as i can tell, StoredMap has the potential
to
implement the extra methods in the jdk 1.5
ConcurrentMap interface.

Yes, implementing ConcurrentMap would be really nice,
and not very difficult. The main problem we have
right now is that we do not depend on Java 1.5, so we
could not actually implement the 1.5 ConcurrentMap
interface.

We could add the methods, however, without
implementing the interface. Are you more interested
in the methods (the functionality) or the
implementation of the standard Java ConcurrentMap
interface?


We're only interested in the functionality, not the interface itself.

Personally, however,
i am more interested in the other methods, all of
which basically do their work only if the current
value matches a given old value. this enables
safe
concurrent updates to a StoredMap even outside of
transactions, basically using an optimistic
scenario:

- read existing key,value (hold no locks)
- copy current value and modify the copy
- write modified value only if current value

matches
what is now in the database (this line within the
context of a RMW lock)

Since we need to write-lock (RMW lock) the record
before writing it, I don't see any advantage to
reading the record without locking initially. We
might as well just:

- read with RMW
- if the current value equals the given value, then
write

This implementation requires the use of a cursor or a
transaction.

any idea if this could be incorporated in the je
codebase anytime soon?

We don't have any plans to do this, but I'm happy to
help you if I can understand better exactly what your
needs are.


well, our basic need is to support concurrent modification of a database, where there is a very small chance of concurrent modification of the same record. (and, before i continue, i'll admit i don't completely understand the relationship between transactions and locking). i can certainly implement this in a transactional environment, but i was hoping to avoid using transactions since the chance of collision is so small. i'm assuming that locking is still used in the non-transactional environment (or else you'd get seriously horked data?), so i was hoping that using concurrentmap-like methods would enable me to concurrently modify a database without transactions, but while still handling the occasional collision. that is why my description of the operations involve the RMW lock only on the write call, so there is no need for cursors (at least, outside of the write method) or transactions. however, i could be misunderstanding the functionality available so please tell me if i'm way off base with what i'm looking for.
greybird

Posts: 1,296
Registered: 07/13/06
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 10:31 AM   in response to: jahlborn in response to: jahlborn
Click to report abuse...   Click to reply to this thread Reply
Hi,

We're only interested in the functionality, not the
interface itself.

OK. Then I'll just explain how to do this using the existing APIs.

well, our basic need is to support concurrent
modification of a database, where there is a very
small chance of concurrent modification of the same
record. (and, before i continue, i'll admit i don't
completely understand the relationship between
transactions and locking). i can certainly implement
this in a transactional environment, but i was hoping
to avoid using transactions since the chance of
collision is so small.

Why not use transactions anyway? Their overhead is very small. If you don't need durability on each operation, call EnvironmentConfig.setTxnNoSync, and you'll see roughly the same performance with transactions as without transactions.

i'm assuming that locking is
still used in the non-transactional environment (or
else you'd get seriously horked data?), so i was

Yes, locking is always used unless you specify READ_UNCOMMITTED (dirty read). Using a cursor (without transactions), the lock is held until the cursor moves or is closed. When using transactions, the lock is held until the transaction ends (abort or commit).

hoping that using concurrentmap-like methods would
enable me to concurrently modify a database without
transactions, but while still handling the occasional
collision. that is why my description of the
operations involve the RMW lock only on the write
call, so there is no need for cursors (at least,
outside of the write method) or transactions.
however, i could be misunderstanding the
functionality available so please tell me if i'm way
off base with what i'm looking for.

If you don't use transactions, then you should use a cursor whenever you need to do an update where you process the existing data in some way -- a read-modify-write cycle. The cursor is needed to hold the lock on the record you're modifying, so that another thread cannot modify it. If you don't use a cursor or transactions, then JE cannot help you to prevent conflicts between multiple threads trying to read-modify-write the same record -- you would have to do that completely on your own.

When reading via a cursor, you have the option of using RMW when you read the record. If you use RMW, this reduces the chance of deadlocks caused when another thread is trying to read-modify-write the same record. However, after you read the record if you decide not to write it, then the RMW lock is wasted and concurrency is reduced unnecessarily.

Therefore, if there is a possibility that multiple threads will read-modify-write the same record, and you always know you will write the record after reading it, use RMW when reading. If you will write it only sometimes, or if there is only one writer thread, then do not use RMW.

If there are multiple writer threads and you do not use RMW, be prepared to handle a deadlock. When a DeadlockException is thrown, you must close all cursors and retry the operation from the beginning. If the operation is very simple -- a single read-modify-write cycle -- this is easy. If there are multiple write operations involved, "undoing" the operation will be difficult and you really should be using transactions. Be aware that if you write to a primary database that has secondary databases configured, multiple write operations are being performed and you cannot easily undo that operation when a deadlock occurs. When using secondaries, the use of transactions is strongly recommended.

All of this is much simpler when using transactions. Please first decide whether you will use transactions or a cursor to perform the read-modify-write cycle. Then if you have questions about how to do this with the collections API, let me know and I can help with that.

Mark
jahlborn

Posts: 34
Registered: 02/12/07
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 11:25 AM   in response to: greybird in response to: greybird
Click to report abuse...   Click to reply to this thread Reply
If you don't use transactions, then you should use a
cursor whenever you need to do an update where you
process the existing data in some way -- a
read-modify-write cycle. The cursor is needed to
hold the lock on the record you're modifying, so that
another thread cannot modify it. If you don't use a
cursor or transactions, then JE cannot help you to
prevent conflicts between multiple threads trying to
read-modify-write the same record -- you would have
to do that completely on your own.

I may not be explaining myself well here. Like i mentioned before, I'm working from a scenario where write conflicts should be very infrequent. thus, i'm hoping to use an optimistic strategy, similar to what i would use with a concurrentmap. the way i usually update a concurrent map (using an optimistic strategy):

- read the existing value (map.get()). after this call, you hold no locks (so someone could modify the value before you write, but i'm assuming infrequent conflicts)
- create new value (maintaining handle to existing value)
- attempt to write new value only if existing value is unchanged (map.replace(key, old, new)) (if this call fails, you loop back to the first step and try again)

this is obviously sub-optimal if you expect frequent, colliding updates. however, we are not. and, the nice part about this api is that the caller does not need to do any locking or transaction management.

When reading via a cursor, you have the option of
using RMW when you read the record. If you use RMW,
this reduces the chance of deadlocks caused when
another thread is trying to read-modify-write the
same record. However, after you read the record if
you decide not to write it, then the RMW lock is
wasted and concurrency is reduced unnecessarily.

Therefore, if there is a possibility that multiple
threads will read-modify-write the same record, and
you always know you will write the record after
reading it, use RMW when reading. If you will write
it only sometimes, or if there is only one writer
thread, then do not use RMW.


we will always be writing the record after we read it. your reasoning definitely makes sense. however, i'm working with the collections api, and i do not see any obvious way to do this. am i missing something?

If there are multiple writer threads and you do not
use RMW, be prepared to handle a deadlock. When a
DeadlockException is thrown, you must close all
cursors and retry the operation from the beginning.
If the operation is very simple -- a single
read-modify-write cycle -- this is easy. If there
are multiple write operations involved, "undoing"
the operation will be difficult and you really
should be using transactions. Be aware that if you
write to a primary database that has secondary
databases configured, multiple write operations are
being performed and you cannot easily undo that
operation when a deadlock occurs. When using
secondaries, the use of transactions is strongly
recommended.

All of this is much simpler when using transactions.
Please first decide whether you will use
transactions or a cursor to perform the
read-modify-write cycle. Then if you have questions
about how to do this with the collections API, let
me know and I can help with that.


the reason that the concurrentmap interface is attractive for someone using the collections api (which we are), is that all that complexity is hidden behind the api. and, by adding these methods, you allow users of the collections api to make safe updates without having to delve into transactions. we don't need to group a bunch of updates together nor update secondary databases or i would definitely use transactions. we just want to make safe, independent updates to single records in a primary database. again, i could certainly use transactions for this, but i'm using the collections api, and was hoping to get away with just that.
greybird

Posts: 1,296
Registered: 07/13/06
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 12:26 PM   in response to: jahlborn in response to: jahlborn
Click to report abuse...   Click to reply to this thread Reply
- read the existing value (map.get()). after this
call, you hold no locks (so someone could modify
the value before you write, but i'm assuming
infrequent conflicts)
- create new value (maintaining handle to existing
value)
- attempt to write new value only if existing value
is unchanged (map.replace(key, old, new)) (if this
call fails, you loop back to the first step and try
again)

This has almost no performance advantage over using a transaction or a cursor. No matter what you do, a write lock must be taken when you update the record. So you might as well take the lock when you read the record. Delaying the lock for a very small amount of time will have almost no advantage.

If you do performance measurements and you find that holding an RMW (write) lock for this extra interval is a performance problem, the solution is to not use RMW. If you do not use RMW, then a read lock (not a write lock) will be taken when reading the record, which will avoid reducing concurrency. When you write, you may get a DeadlockException but this will be extremely rare, since you say there is very little chance of conflicts among writers.

Another way of saying this is: If we were to implement this API, we would have to use a cursor, and lock the record when we read it. Therefore, this API (while perhaps convenient) has no performance advantage.

These are the factors from a performance perspective, not an API perspective. More on the API below.

the reason that the concurrentmap interface is
attractive for someone using the collections api
(which we are), is that all that complexity is hidden
behind the api. and, by adding these methods, you
allow users of the collections api to make safe
updates without having to delve into transactions.
we don't need to group a bunch of updates together
nor update secondary databases or i would definitely
use transactions. we just want to make safe,
independent updates to single records in a primary
database. again, i could certainly use transactions
for this, but i'm using the collections api, and was
hoping to get away with just that.

I don't understand why using transactions is difficult or undesirable, it's quite easy, especially for the operation you describe. Can you explain further?

In any case, if you do not use transactions and you want to use the collections API, then you have two options:

1) If you want the concurrent map interface or new methods that are equivalent, you'll have to extend the source code yourself to get this capability. Adding this is a good idea as a convenience method for non-transactional use. However, because this functionality is currently available and easy to use with transactions, it will probably not be a high priority for us to add this, although we'll definitely keep your request in mind during future planning and we'll watch for other users that request this. It may also be important to add support for the ConcurrentMap interface, for interface compatibility reasons, in the future; but so far, we have not had any requests for this. I described earlier how this extension to StoredMap could be implemented using cursors, if you decide to do it yourself.

2) To use the current collections API to do a read-write-modify without transactions you can do the following. A StoredIterator is effectively a cursor. To use RMW, call StoredIterator.setReadModifyWrite(true).
Object myKey;
StoredSortedMap myMap;
StoredSortedMap subMap = (StoredSortedMap) myMap.sublMap(myKey, true, myKey, true);
StoredIterator i = subMap.storedIterator();
try {
if (i.hasNext()) {
Object val = i.next();
// do some processing on val
i.set(val);
}
} finally {
i.close();
}

Please let me know what you think.

Mark
jahlborn

Posts: 34
Registered: 02/12/07
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 12:54 PM   in response to: greybird in response to: greybird
Click to report abuse...   Click to reply to this thread Reply
This has almost no performance advantage over using a
transaction or a cursor. No matter what you do, a
write lock must be taken when you update the record.
So you might as well take the lock when you read the
record. Delaying the lock for a very small amount
of time will have almost no advantage.

i completely understand.

I don't understand why using transactions is
difficult or undesirable, it's quite easy, especially
for the operation you describe. Can you explain
further?

to expand further on our usage, bdb code in our system is actually hidden behind our own "persistent collection" api. currently, this api exposes no transaction related stuff. in order to utilize transactions in our client code, we would have to create some sort of generic transaction api for our persistent collection api.

as for hiding the transaction behind a method implementation, perhaps you could explain for me the implications of opening a database (and environment for that matter) as "transactional" vs "non-transactional". in order to utilize transactions within certain methods behind our persistent collection api, i would have to open every env/db as "transactional". however, i was assuming there was some sort of overall performance implication in this (otherwise, why have the option?). i mean, if the system will perform the same way if i just open every env/db as transactional, i could easily implement the map.replace() method using transactions instead of using the method you include below.

In any case, if you do not use transactions and you
want to use the collections API, then you have two
options:

1) If you want the concurrent map interface or new
methods that are equivalent, you'll have to extend
the source code yourself to get this capability.
Adding this is a good idea as a convenience method
for non-transactional use. However, because this
functionality is currently available and easy to use
with transactions, it will probably not be a high
priority for us to add this, although we'll
definitely keep your request in mind during future
planning and we'll watch for other users that
request this. It may also be important to add
support for the ConcurrentMap interface, for
interface compatibility reasons, in the future; but
so far, we have not had any requests for this. I
described earlier how this extension to StoredMap
could be implemented using cursors, if you decide to
do it yourself.

2) To use the current collections API to do a
read-write-modify without transactions you can do the
following. A StoredIterator is effectively a cursor.
To use RMW, call
StoredIterator.setReadModifyWrite(true).
pre]
Object myKey;
StoredSortedMap myMap;
StoredSortedMap subMap = (StoredSortedMap)
myMap.sublMap(myKey, true, myKey, true);
StoredIterator i = subMap.storedIterator();
try {
if (i.hasNext()) {
Object val = i.next();
// do some processing on val
i.set(val);
}
} finally {
i.close();
/pre]


ah, beautiful. i didn't realize you could enable RMW through the use of a storediterator. that should provide a very simple implementation of concurrentmap.replace() without the use of transactions. which brings me back to my question above. does enabling transactions by default affect overall performance?
greybird

Posts: 1,296
Registered: 07/13/06
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 1:09 PM   in response to: jahlborn in response to: jahlborn
Click to report abuse...   Click to reply to this thread Reply
Hi,

In BDB JE, enabling transactions adds no overhead when an environment is opened. The reason for the EnvironmentConfig.setTransactional API is to enable the transaction feature For our commercial customers (we have a dual license product) we sell the transactional version of the product separately from the non-transactional version -- they are two products. Whether transactions are configured or not is the distinguishing factor.

When each transaction is committed, a very small commit record is written to the log -- I doubt that will have an impact on performance for you.

The main overhead for transactions is to provide durability. Without transactions, durability is only guaranteed when you call Environment.sync -- at that time, we checkpoint, flush all buffers and fsync to disk.

With transactions, we flush and fsync at every commit. This is very expensive and is required for strict durability.

But if you are using transactions and you don't need that durability, you can simply call EnvironmentConfig.setTxnNoSync(true) and the durability (and overhead) with transactions will be the same as without transactions. In this case, you get the atomicity of transactions without the durability.

Note that when you configure a database as transactional, we will use auto-commit if you don't explicitly begin a transaction. So you don't have to change your operation code except in cases where you want to use a transaction, such as for this read-modify-write cycle.

Mark
jahlborn

Posts: 34
Registered: 02/12/07
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 1:45 PM   in response to: greybird in response to: greybird
Click to report abuse...   Click to reply to this thread Reply
But if you are using transactions and you don't need
that durability, you can simply call
EnvironmentConfig.setTxnNoSync(true) and the
durability (and overhead) with transactions will be
the same as without transactions. In this case, you
get the atomicity of transactions without the
durability.

okay, that makes a lot more sense now. i actually played a bit already with transactions and the various levels of durability, and that is pretty much what i saw in my limited tests. but, i couldn't figure out why transactions still had to be explicitly enabled. you might want to add that to the docs somewhere so others don't come to the same erroneous conclusions that i did.

So anyway, working from your example, i've implemented 3 of the 4 concurrentmap methods. however, putIfAbsent() eludes me as the collections api does not seem to expose a way to call this on the underlying database? This is what i have so far, does it look reasonable?:

  private StoredIterator getIteratorForKey(Object key)
  {
    StoredSortedMap subMap = (StoredSortedMap)
      ((StoredSortedMap)_map).subMap(key, true, key, true);
    return ((StoredEntrySet)subMap.entrySet()).storedIterator();
  }
  
  public V putIfAbsent(K key, V value)
  {
    // FIXME
    return null;
  }
 
  public boolean remove(Object key, Object value)
  {
    StoredIterator iter = getIteratorForKey(key);
    try {
      iter.setReadModifyWrite(true);
      if(!iter.hasNext()) {
        return false;
      }
      Map.Entry e = (Map.Entry)iter.next();
      Object curVal = e.getValue();
      if(!ObjectUtils.equals(value, curVal)) {
        // was modified
        return false;
      }
      // good to go (we have the write lock already)
      iter.remove();
    } finally {
      iter.close();
    }
    return true;
  }
 
  @SuppressWarnings("unchecked")
  public boolean replace(K key, V oldValue, V newValue)
  {
    StoredIterator iter = getIteratorForKey(key);
    try {
      iter.setReadModifyWrite(true);
      if(!iter.hasNext()) {
        return false;
      }
      Map.Entry e = (Map.Entry)iter.next();
      Object curVal = e.getValue();
      if(!ObjectUtils.equals(oldValue, curVal)) {
        // was modified
        return false;
      }
      // set new value (we have the write lock already)
      e.setValue(newValue);
    } finally {
      iter.close();
    }
    return true;
  }
 
  @SuppressWarnings("unchecked")
  public V replace(K key, V value)
  {
    Object curValue = null;
    StoredIterator iter = getIteratorForKey(key);
    try {
      iter.setReadModifyWrite(true);
      if(iter.hasNext()) {
        Map.Entry e = (Map.Entry)iter.next();
        curValue = e.getValue();
        // set new value (we have the write lock already)
        e.setValue(value);
      }
    } finally {
      iter.close();
    }
    return (V)curValue;
  }
greybird

Posts: 1,296
Registered: 07/13/06
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 19, 2007 4:32 PM   in response to: jahlborn in response to: jahlborn
Click to report abuse...   Click to reply to this thread Reply
okay, that makes a lot more sense now. i actually
played a bit already with transactions and the
various levels of durability, and that is pretty much
what i saw in my limited tests. but, i couldn't
figure out why transactions still had to be
explicitly enabled. you might want to add that to
the docs somewhere so others don't come to the same
erroneous conclusions that i did.

OK, will do.

So anyway, working from your example, i've
implemented 3 of the 4 concurrentmap methods.
however, putIfAbsent() eludes me as the collections
api does not seem to expose a way to call this on
the underlying database? This is what i have so
far, does it look reasonable?:

These look good!

You're correct that putNoOverwrite is not available in the Collections API.

Since you cannot access package-private members, the only way to do this is to implement it at the base API level. You will have to save the Database and bindings that you used to create the StoredMap, and use them to call Database.putNoOverwrite.

Another way is to configure the Serializable isolation level and simply put() if get() returns null, but this requires using transactions.

Mark
jahlborn

Posts: 34
Registered: 02/12/07
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 20, 2007 7:09 AM   in response to: greybird in response to: greybird
Click to report abuse...   Click to reply to this thread Reply
You're correct that putNoOverwrite is not available
in the Collections API.

Since you cannot access package-private members, the
only way to do this is to implement it at the base
API level. You will have to save the Database and
bindings that you used to create the StoredMap, and
use them to call Database.putNoOverwrite.


First, thanks for all your help with this!

So, in an effort to see if i could write these methods "correctly", i dove into the lower level api. I ended up reimplementing the previous methods and implementing the remaining method. (for one thing, the previous implementations did not work if transactions were enabled for the db). here is my final impl. i'm not quite sure about the putifabsent() method (don't know all the scenarios that might be encountered with/without transactions). how does this look (and, if reasonable, could it be rolled into existing codebase)?:

public class StoredConcurrentMap extends StoredMap
{
 
  public StoredConcurrentMap(
      Database database, EntryBinding keyBinding,
      EntryBinding valueEntityBinding, boolean writeAllowed)
  {
    super(database, keyBinding, valueEntityBinding, writeAllowed);
  }
 
  public Object putIfAbsent(Object key, Object value)
  {
    Object oldValue = null;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true);
      while(true) {
        if(OperationStatus.SUCCESS ==
           cursor.putNoOverwrite(key, value, false)) {
          // we succeeded
          break;
        } else if(OperationStatus.SUCCESS ==
                  cursor.getSearchKey(key, null, false)) {
          // someone else beat us to it
          oldValue = cursor.getCurrentValue();
          break;
        }
 
        // we couldn't put and we couldn't get, try again
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return oldValue;
  }
 
  public boolean remove(Object key, Object value)
  {
    boolean removed = false;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        Object curValue = cursor.getCurrentValue();
        if(ObjectUtils.equals(curValue, value)) {
          cursor.delete();
          removed = true;
        }
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return removed;
  }
 
  public boolean replace(Object key, Object oldValue, Object newValue)
  {
    boolean replaced = false;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        Object curValue = cursor.getCurrentValue();
        if(ObjectUtils.equals(curValue, oldValue)) {
          cursor.putCurrent(newValue);
          replaced = true;
        }
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return replaced;
  }
 
  public Object replace(Object key, Object value)
  {
    Object curValue = null;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        curValue = cursor.getCurrentValue();
        cursor.putCurrent(value);
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return curValue;
  }
}
greybird

Posts: 1,296
Registered: 07/13/06
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Feb 21, 2007 4:11 PM   in response to: jahlborn in response to: jahlborn
Click to report abuse...   Click to reply to this thread Reply
Sorry the slow reply.

First, thanks for all your help with this!

You're welcome, thanks for your effort and ideas!

So, in an effort to see if i could write these
methods "correctly", i dove into the lower level api.
I ended up reimplementing the previous methods and
implementing the remaining method. (for one thing,
the previous implementations did not work if
transactions were enabled for the db). here is my
final impl. i'm not quite sure about the
putifabsent() method (don't know all the scenarios
that might be encountered with/without
transactions). how does this look (and, if
reasonable, could it be rolled into existing
codebase)?:

Thanks again for these ideas. We'll go ahead and add an implementation of the ConcurrentMap methods for a future release.

I can't see anything wrong with your implementation. The loop you're using for putIfAbsent is correct -- it works with and without transactions. With transactions, the loop is not necessary if the Serializable isolation level is configured, but this is not the default -- what you've done should work in all cases.

Mark
jahlborn

Posts: 34
Registered: 02/12/07
Re: thoughts on adding ConcurrentMap methods to StoredMap?
Posted: Mar 29, 2007 11:45 AM   in response to: greybird in response to: greybird
Click to report abuse...   Click to reply to this thread Reply
hey,
just wanted to follow up on this a little. after using the concurrent implementation for a while (with transactions enabled), i experimented with extending the api to add a "transaction-like" extension to the concurrentmap api. this basically blends the concurrentmap api with the option of better performance when transactions are enabled in the underlying bdb. there are three methods added, which basically allow for "locking" a given key and "committing" or "rolling back" the updates as appropriate (if transactions are not enabled, the new methods are largely no-ops, and you get normal concurrent map behavior). thought i'd post the new code in case you were interested in providing this facility to other users (or, you could just stick with the original version posted above).

Example usage:
Object key;
try {
  Object value = map.getForUpdate(key);
  // ... run concurrent ops on value ...
  map.replace(key, value, newValue);
  map.completeUpdate(key);
} finally {
  map.closeUpdate(key);
}


New implementation:
public class StoredConcurrentMap extends StoredMap
{
  private final ThreadLocal<UpdateState> _updateState =
    new ThreadLocal<UpdateState>();
 
  public StoredConcurrentMap(
      Database database, EntryBinding keyBinding,
      EntryBinding valueEntityBinding, boolean writeAllowed)
  {
    super(database, keyBinding, valueEntityBinding, writeAllowed);
  }
  
  public Object getForUpdate(Object key)
  {
    if(_updateState.get() != null) {
      throw new IllegalStateException(
          "previous update still outstanding for key " +
          _updateState.get()._key);
    }
    UpdateState updateState = new UpdateState(key, beginAutoCommit());
    _updateState.set(updateState);
    
    DataCursor cursor = null;
    Object value = null;
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        // takeover ownership of the cursor
        value = updateState.loadCurrentValue(cursor);
        cursor = null;
      }
      closeCursor(cursor);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, false);
    }
    return value;
  }
 
  public void completeUpdate(Object key)
  {
    UpdateState updateState = getUpdateState(key);
    if(updateState != null) {
      try {
        updateState.clearCurrentValue(this);
        commitAutoCommit(updateState._doAutoCommit);
        // only clear the reference if everything succeeds
        _updateState.set(null);
      } catch(DatabaseException e) {
        throw new RuntimeExceptionWrapper(e);
      }
    }
  }
 
  public void closeUpdate(Object key)
  {
    UpdateState updateState = getUpdateState(key);
    if(updateState != null) {
      // this op failed, abort (clear the reference regardless of what happens
      // below)
      _updateState.set(null);
      try {
        updateState.clearCurrentValue(this);
        view.getCurrentTxn().abortTransaction();
      } catch(DatabaseException ignored) {
      }
    }
  }
  
  public Object putIfAbsent(Object key, Object value)
  {
    UpdateState updateState = getUpdateState(key);
    if(valueExists(updateState)) {
      return updateState._curValue;
    }
    
    Object oldValue = null;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true);
      while(true) {
        if(OperationStatus.SUCCESS ==
           cursor.putNoOverwrite(key, value, false)) {
          // we succeeded
          break;
        } else if(OperationStatus.SUCCESS ==
                  cursor.getSearchKey(key, null, false)) {
          // someone else beat us to it
          oldValue = cursor.getCurrentValue();
          break;
        }
 
        // we couldn't put and we couldn't get, try again
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return oldValue;
  }
 
  public boolean remove(Object key, Object value)
  {
    UpdateState updateState = getUpdateState(key);
    if(valueExists(updateState)) {
      if(ObjectUtils.equals(updateState._curValue, value)) {
        try {
          updateState._cursor.delete();
          updateState.clearCurrentValue(this);
          return true;
        } catch (Exception e) {
          throw handleException(e, false);
        }
      } else {
        return false;
      }
    }
 
    boolean removed = false;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        Object curValue = cursor.getCurrentValue();
        if(ObjectUtils.equals(curValue, value)) {
          cursor.delete();
          removed = true;
        }
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return removed;
  }
 
  public boolean replace(Object key, Object oldValue, Object newValue)
  {
    UpdateState updateState = getUpdateState(key);
    if(valueExists(updateState)) {
      if(ObjectUtils.equals(updateState._curValue, oldValue)) {
        try {
          updateState.replaceCurrentValue(newValue);
          return true;
        } catch (Exception e) {
          throw handleException(e, false);
        }
      } else {
        return false;
      }
    }
    
    boolean replaced = false;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        Object curValue = cursor.getCurrentValue();
        if(ObjectUtils.equals(curValue, oldValue)) {
          cursor.putCurrent(newValue);
          replaced = true;
        }
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return replaced;
  }
 
  public Object replace(Object key, Object value)
  {
    UpdateState updateState = getUpdateState(key);
    if(valueExists(updateState)) {
      try {
        return updateState.replaceCurrentValue(value);
      } catch (Exception e) {
        throw handleException(e, false);
      }
    }
 
    Object curValue = null;
    DataCursor cursor = null;
    boolean doAutoCommit = beginAutoCommit();
    try {
      cursor = new DataCursor(view, true, key);
      if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
        curValue = cursor.getCurrentValue();
        cursor.putCurrent(value);
      }
      closeCursor(cursor);
      commitAutoCommit(doAutoCommit);
    } catch (Exception e) {
      closeCursor(cursor);
      throw handleException(e, doAutoCommit);
    }
    return curValue;
  }
 
  
  /**
   * @return the current UpdateState for the given key, if any, {@code null}
   *         otherwise.
   */
  private UpdateState getUpdateState(Object key)
  {
    UpdateState updateState = _updateState.get();
    if((updateState != null) && (ObjectUtils.equals(updateState._key, key))) {
      return updateState;
    }
    return null;
  }
  
  /**
   * @return {@code true} if the update state exists and found a value (which
   *         is currently locked)
   */
  private boolean valueExists(UpdateState updateState)
  {
    return((updateState != null) && (updateState._cursor != null));
  }
 
 
  /**
   * Maintains state about an object loaded in a
   * {@link StoredConcurrentMap#getForUpdate} call.
   */
  private static class UpdateState
  {
    public final Object _key;
    public final boolean _doAutoCommit;
    public DataCursor _cursor;
    public Object _curValue;
 
    private UpdateState(Object key, boolean doAutoCommit) {
      _key = key;
      _doAutoCommit = doAutoCommit;
    }
 
    /**
     * Loads the current value from the given cursor, and maintains a
     * reference to the given cursor and the loaded value.
     */
    public Object loadCurrentValue(DataCursor cursor)
      throws DatabaseException
    {
      _cursor = cursor;
      _curValue = _cursor.getCurrentValue();
      return _curValue;
    }
 
    /**
     * Replaces the current value in the current cursor with the given value.
     * @return the old value
     */
    public Object replaceCurrentValue(Object newValue)
      throws DatabaseException
    {
      Object oldValue = _curValue;
      _cursor.putCurrent(newValue);
      _curValue = newValue;
      return oldValue;
    }
 
    /**
     * Closes the current curstor and clears the references to the cursor and
     * current value.
     */
    public void clearCurrentValue(StoredConcurrentMap map) {
      try {
        map.closeCursor(_cursor);
      } finally {
        _cursor = null;
        _curValue = null;
      }
    }
    
  }
  
}
Legend
Guru Guru : 2500 - 1000000 pts
Expert Expert : 1000 - 2499 pts
Pro Pro : 500 - 999 pts
Journeyman Journeyman : 200 - 499 pts
Newbie Newbie : 0 - 199 pts
Oracle ACE Director
Oracle ACE Member
Oracle Employee ACE
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums