Home » Blogs » R. Tyler Croy's blog

Lazily loading attributes.

September 14, 2008 - 11:27pm | by R. Tyler Croy

I found myself talking to Jason today about the virtues of getattr(), setattr(), and hasattr() in Python and "abusing" the dynamic nature of the language which reminded me of some lazy-loading code I wrote a while back. In February I found the need to have portions of the logic behind one of our web applications fetch data once per-request. The nature of the web applications we're building on top of the MySpace, Hi5 and Facebook platforms require some level of network data-access (traditionally via REST-like APIs). This breaks our data access model into the following tiers:

Dia FTW

Working with network-centric data resources is difficult in any scenario (desktop, mobile, web) but the particularly difficult thing about network data access in the mod_python-driven request model is that it will be synchronous (mod_python doesn't support "asynchronous pages" like ASP.NET does). This means every REST call to Facebook, for example, is going to block execution of the request handler until the REST request to Facebook's API tier completes. 
  1.  def request_handler(self, *args, **kwargs):
  2.     fb_uid = kwargs.get('fb_sig_user')
  3.     print "Fetching the name for %s" % fb_uid
  4.     print time.time()
  5.     name = facebook.users.getInfo(uid=fb_uid) 
  6.     ### WAIT-WAIT-WAIT-WAIT-WAIT
  7.     print time.time()      
  8.     ### Continue generating the page...
There is also a network hit (albeit minor) for accessing cached data or data stored in databases. The general idea is that we'll need to have some level of data resident in memory through-out a request that can differ widely from request-to-request.

Lazy loading in Python

To help avoid unnecessary database access or network access I wrote a bit of class-sugar to make this a bit easier and more fail-proof:

  1. class LazyProgrammer(object):
  2.     ''' 
  3.         LazyProgrammer allows for lazily-loaded attributes on the subclasses 
  4.         of this object. In order to enable lazily-loaded attributes define
  5.         "_X_attr_init()" for the attribute "obj.X"
  6.     '''
  7.     def __getattr__(self, name):
  8.         rc = object.__getattribute__(self, '_%s_attr_init')()
  9.         setattr(self, name, rc) 
  10.         return rc
This makes developing with network-centric web applications a bit easier, for example, if I have a "friends" lazily-loading attribute off the base "FacebookRequest" class, all developers writing code subclassing FacebookRequest can simply refer to self.friends and feel confident they aren't incurring unnecessary bandwidth hits, and the friends-list fetching code is located in once spot. If one-per-request starts to become too resource intensive as well, it'd be trivial to override the _friends_attr_init() method to hit a caching server instead of the REST servers first, without needing to change any code "downstream."

Lazy loading in C#

Since C# is not a dynamically-typed language like Python or JavaScript, you can't implement lazily-loaded attributes in the same fashion (calling something like setattr()) but you can "abuse" properties in a manner similar to the C# singleton pattern, to get the desired effect:

  1. using System;
  2. using System.Collections.Generic;
  3.  
  4. public class LazySharp
  5. {
  6. 	#region "Lazy Members"
  7. 	private Dictionary<int, string> _names = null;
  8. 	#endregion
  9.  
  10. 	#region "Lazy Properties"
  11. 	public Dictionary<int, string> Names
  12. 	{
  13. 		get {
  14. 			if (this._names == null)
  15. 				this._names = this.SomeExpensiveCall();
  16. 			return this._names;
  17. 		}
  18. 	}
  19. 	#endregion
  20. }
Admittedly I don't find myself writing Facebook/MySpace/Hi5 applications these days on top of ASP.NET so I cannot say I actually use the class above in production, but conceptually it makes sense.

Lazy loading attributes I find useful in the more hodge-podge situations, where code and feature-sets have both grown organically over time, they're not for everybody but I figured I'd share anyways.

Tags:

Comments

by Anonymous | September 15, 2008 - 11:01am

that's actually a ugly way

that's actually a ugly way for lazy loaded attributes in python

the nice way would be having a non-data descriptor that wraps the method for loading the attribute

that way it doesn't require getattr hacks

by Anonymous | September 15, 2008 - 12:21am

addition

Can be completed with double-lock-thread-safe pattern :

  1. private object _lock = new object();
  2. #
  3. #region "Lazy Properties"
  4. #
  5. public Dictionary<int, string> Names
  6. {
  7.   get {
  8.     if (this._names == null)
  9.       lock(_lock)
  10.       {
  11.         if(this._names == null)
  12.           this._names = this.SomeExpensiveCall();
  13.       }
  14.     return this._names;
  15.   }
  16. }
  17. #endregion

by Anonymous | September 15, 2008 - 4:19am

singleton?

that's just a singleton, and using a double lock really sucks, please refer to this: http://www.yoda.arachsys.com/csharp/singleton.html also try this generic singleton that is also thread safe:

  1. public class Singleton<T> where T : class, new()
  2. {
  3. 	private static readonly T instance = new T(); 
  4.  
  5. 	public static T Instance
  6. 	{
  7. 		get { return instance; }
  8. 	} 
  9.  
  10. 	private Singleton()
  11. 	{ } 
  12. }

by Anonymous | September 15, 2008 - 5:53am

not singleton

No it's not a singleton, it use instance context (not shared by different instances of same type).
The singleton implementation that you refer to rely on static field, can't be used in this case.

by Anonymous | August 21, 2009 - 6:26am

Shower cubicles

Hi This is good code just used it, i have not found any bugs eather so thanks and keep up the good work.

jill xx