Showing posts with label Architecture. Show all posts
Showing posts with label Architecture. Show all posts

Monday, November 16, 2009

TechEd Europe 2009

Well TechEd Europe 2009 held in Berlim, finished yesterday. It was another great experience, where I learned some stuff that I was eager to get my hands on (VS 2010 Architecture), contacted some Portuguese fellows and MS gurus, and discovered a little (not much time available, and at 4PM it’s completely dark) of Berlim. Also received a free copy of Win7 and some kits (SQL 2008, Win 2008), which will come handy.

There were also some missteps:

  • On arrival, all of the bags that came from Lisbon didn’t make it to the airplane and were left in Madrid. Not very pleasant, arriving at 22:45 at Berlim and not having any more clothes when the temperature is 2ºC.
  • The wireless/Internet on the hotel was not free, and very expensive: 9,95 €/hour, or 19,95 €/day.
  • The logistics around the event had some problems, namely with the check-in, and the cloakroom (huge lines). But, for a first event (for Berlim) and with > 7.000 people attending, it sould be expected. According to some powerpoints of the sessions, there will be no DVD with the sessions recordings and powerpoints available for the attendees (but the e-mail with the TechEd offers, mentions the MMS 2009 DVDs Breakout recordings and HOLs, so I don’t know what to make of it).
  • Almost nobody speaks English in Berlim, so we were on our own. The younger people speak a little English (but one person from the hotel confused “no problem” with “not possible”, making for a very strange conversation).
  • The flight back was at 7:30 AM so waking up and checking-out between 4:30 AM and 5:30 AM was a bit tough. Also some confusions with the credit card payment of the hotel (it was just reserved, apparently), made for a tought day (not over yet).
  • The connection flight is only at 15:45, so we have to wait out in the aeroport for more than 4 hours and are already very tired. Also the WI-FI at the airport is not free, so no Internet.

But back to the event. Microsoft grouped the IT sessions with the Developer sessions, which has its pros and cons. On the upside, I was able to assist some very good and on-time sessions on “Virtualization and Consolidation” (Gopal Ashok), “Upgrading to SQL 2008” (Dandy Wein), and two very good sessions from Russinovich (from SysInternals fame) “Pushing the limits of Windows”, “Windows Troubleshooting”.

On the downside, the keynote was more IT oriented (Exchange 2010, a new efficiency debate – which was a bit boring, and Windows 2008 R2 and Hyper-V).

For me, the best sessions on the Architecture Track were David Chappell “IT Innovation”, Rafal Lucawiecki “Predictive Programming”, and Don Smith “Application Architecture Guide”. On the Development Track, I really enjoyed the ones around VS2010 Architecture Tools: Peter Provost “Architecture Discovery and Validation”, and Doug Seven “TFS: Become productive in under 30 minutes”.

A bit of a let down were the sessions from Tess Fernandez “Debugging ASP.NET” and Brian Keller “SW Testing in VS 2010”. Not that they were bad, on the contrary, but I was expecting more (maybe it was the expectations).

Update:

The post was written on Saturday, 14 November on the airport, so the event finished 13 November. The post was done today because there was no free wireless on the airport and the remainder of the weekend was just family time.

Marcas Technorati: ,,,

Tuesday, January 27, 2009

Top 100 Books of SW Engineering

I just found out a great compilation of the top 100 books of Software Engineering. The method for computing the list involves Amazon ratings, Jolt Awards and Google hits, so it's as scientific as it can get...

Looking at the list makes for a tremendous wish-list/bookshelf:

#
Author(s) / Title

1
Steve McConnell

Code Complete: A Practical Handbook of Software Construction (2nd Edition)

2
Elisabeth Freeman, etc.

Head First Design Patterns

3
Steve McConnell

Rapid Development

4
Erich Gamma

Design Patterns: Elements of Reusable Object-Oriented Software

5
Bruce Schneier

Applied Cryptography: Protocols, Algorithms, and Source Code (2nd Edition)

6
Robert C. Martin

Agile Software Development: Principles, Patterns and Practices

7
Joel Spolsky

Joel on Software

8
Tom DeMarco, Timothy Lister

Peopleware: Productive Projects and Teams (2nd Edition)

9
Frederick P. Brooks

The Mythical Man-Month, Anniversary Edition (2nd Edition)

10
Martin Fowler

Refactoring: Improving the Design of Existing Code

11
Mike Cohn

Agile Estimating and Planning

12
Alistair Cockburn

Writing Effective Use Cases

13
Bertrand Meyer

Object-Oriented Software Construction (2nd Edition)

14
Steve McConnell

Software Estimation: Demystifying the Black Art

15
Mike Cohn

User Stories Applied: For Agile Software Development

16
Donald E. Knuth

The Art of Computer Programming, Volumes 1-3 Boxed Set (2nd Edition)

17
Martin Fowler

Patterns of Enterprise Application Architecture

18
Jeffrey Friedl

Mastering Regular Expressions

19
Andrew Hunt, David Thomas

The Pragmatic Programmer: From Journeyman to Master

20
Karl E. Wiegers

Software Requirements (2nd Edition)

The full article is at Top 100 Best Software Engineering Books, Ever.

Friday, December 12, 2008

Architecture again

Right after my last post (which I was on my mind because it happened to me a couple of times this year and the last year), I discovered (thanks to a post from a friend patterns & practices- App Arch Guide Pocket Guides) that MS PnP has some interesting stuff on this topic (shame for not looking recently on PnP).

Apart from the pocket guides, that I haven't looked into, what I found were some cheat sheets and diagrams.

PnP Application Architecture Frame Cheat Sheet describes architecture frames (authentication, caching, ... and 14 others), quality attributes (14) and the mapping between application types, architecture styles and the architecture frames (and the common issues). It's great stuff, and should be mandatory read for every developper. Shame that Microsoft only has teaching offerings around products (maybe that's what the industry cares about).

ArchMetaFrame

Another one is PnP Application Type Matrix Cheat Sheet that summarizes the main application types, their benefits, considerations, scenarios and solutions.

The last one is PnP Visio index Diagrams that MS makes available for modification and use to document our architectures. Even if only used for proposals, it's very good and allows reuse.

Marcas Technorati: ,

Friday, December 05, 2008

Architecture & Process

Lately, I've been surprised by how little attention some developers pay to application architecture. By architecture here, I mean to say the high level decisions that crosscut all the application code.

Recently, during the transfer of an application from another team, I was shocked that there was no clear vision of the main decisions regarding some fundamental aspects of a web application (I don't mean documented, just a clear definition of the decision or the approaches taken in the implementation). Every web application should make explicit the major decisions (and the reasioning behind them) regarding:

    • Logging (policy - how, when; and tools/code)
    • Exception Handling
    • Data Acess (ADO.NET, ORM, Linq)
        • Transaction Management (sharing the same transaction, creating and commiting transactions)
    • Session Management (asp.net memory, DB, custom)
    • Security (low-level - sql injection, cross scripting; high-level - isolate data between different users/entitys/geographically)
    • Profiles and Permissions (management of users/groups and the corresponding permissions on application functionalities)
    • Operation Auditing (especially in financial systems)
    • Composition (tiers, layers, service oriented)
    • Dependencies (3rd party tools, components, services)
    • Patterns used (MVC, Singleton, Composite)
    • Naming conventions
    • Configuration management (reference tables, configuration values, connection strings, etc)
    • Concurrency (synchronizatios, async callbacks, threading)
    • ...

 

There's also other important stuff, more on the process/principles side, that gain by being defined:

    • Planning, prioritization & risk management
    • Organization (teams, projects)
    • Tools
    • Automation of tasks (building, testing, importing reference data)
    • Testing and Coverage (unit testing, integration testing, coverage of significant program states)
    • Refactoring
    • Documentation (design, architecture, glossary, major entities, workarounds to problems)
    • Bug/Incident tracking
    • Version Control (tool, policy)
    • Communication
    • DRY
    • Responsabilities (code & people) and Separation of concerns
    • Coupling & Cohesion
    • Done?

 

As with everything, there's also the risk of overdoing stuff, or doing it as an end (and not as mean to an end).

If just 1/3 of the list were implemented, maintenance would be such an easier job...

Marcas Technorati: ,

Wednesday, September 19, 2007

The Past and the Future

Just finished reading two great articles about the past, history, and the future.

The first article is from Joel Spolsky (Strategy Letter VI) and how history tends to repeat itself. Starting there, and with some extrapolation from performance considerations in the past (with Excel) and the present (with Ajax and JavaScript), some patterns become apparent (although the actual names could change). But the real kicker is from the comparison between mainframes, CICS and the evolution to microcomputers and Windows to the current state of the Web and it's problems (interoperability between Ajax applications). Very good and thought provoking. It just reminds me of how circular history is: we started with mainframes and dumb terminals, then to client PCs applications (Office, Access, fat clients), then back to big servers and thin clients (with the Web), and with smart clients we are returning the processing power to the client.

 

The other great post (more of a link to actual MSDN content - Finally some news on PLINQ), reveals the future after .NET 3.5 and MS solutions to the increasing cores with the same speed problem.

The MSDN article shows the primitives for concurrency and how easily concurrency can be "added" to a program. The PTasks article shows it's application to Linq (specially Linq to SQL).

Just checked the MSDN October cover, and it's mainly about concurrency/threads and multiple cores. I'll put it on my short list of "must read"s.

Marcas Technorati: , ,

Friday, September 14, 2007

Yet another PK debate

Recently the article in http://weblogs.sqlteam.com/jeffs/archive/2007/08/23/composite_primary_keys.aspx got me thinking again in DB design, especially the PK dilemma (natural or surrogate).

So i googled around to see the current discussion and came up with some links:

Articles defending surrogate keys:
http://www.4guysfromrolla.com/webtech/110806-1.shtml
http://searchsqlserver.techtarget.com/general/0,295582,sid87_gci1156604,00.html
http://r937.com/20020620.html
http://www.bcarter.com/intsurr1.htm

http://www.aspfaq.com/show.asp?id=2504
http://databases.aspfaq.com/database/what-should-i-choose-for-my-primary-key.html
Articles defending natural keys:
http://www.sqlteam.com/article/identity-and-primary-keys

http://www.solidqualitylearning.com/blogs/Dejan/archive/2005/02/24/434.aspx

http://groups.google.com/group/microsoft.public.sqlserver.programming/msg/6d61dbf80d6f0fb6?hl=en&lr=&ie=UTF-8&oe=UTF-8&rnum=14

But returning to the initial article, in it, it's argued that in relation tables the PK should be natural (composite of the PKs of the relations). Naturally, it seems right, and it aligns well with the relational theory. Unfortunately, business clients don't align so well ;)

The initial problem is modeling a CustomerProducts relation table, to which the following two solutions are proposed:

Create table CustomerProducts
(
    CustomerID int references Customers(CustomerID) not null,
    ProductID int references Products(ProductID) not null,
    OrderLimit int not null,
    Primary key (CustomerID, ProductID)
)

Create table CustomerProducts
(
    Customer_ProductID int identity primary key,
    CustomerID int references Customers(CustomerID) not null,
    ProductID int references Products(ProductID) not null,
    OrderLimit int not null
)

The exchange of ideas in the comments was good, and it got me thinking how I do design.

So here are my thoughts (published also in the comments of the initial article).

One of the downsides of the modeling based on natural keys is it's averse to change.

Of course with an identity column, the model, the application must be validated, and possibly changed (including migration) to acommodate the change. But the change doesn't ripple throughout the model as with natural keys.

I honestly can't see the big difference between entity tables and relation tables. I could argue (and many DB purists would) that the PK in an entity table should be a natural key (probably composite).

And it appears you can relate to the problems it introduces:

  • very difficult to change (I can only imagine what must have been the migration of a DB where ISBN was used as a PK, or using the bank filial as part of the PKs of bank cards - imagine changing the account);
  • no significant advantadge over the more flexible surrogate model.

To me the same argument holds with relation tables. It eases future changes. The disadvantages are extra complexity, but the end result is more flexible.

I haven't seen a solid example of the supposed advantages (in practice) of the composite key approach. The extra space isn't so much as stated if considering that other indexes include the PK. So the wider is the PK, bigger becomes the index.

I haven't seen any examples or papers that demonstrate in practice that it's more performant to use composite keys (in general).

Regarding the purity or correctness of the model, I can only say, that it's a model... It can be abused if one tries to be to flexible, but sometimes it's better to be flexible and pragmatic than to be rigid and theoretic.

This comment generated a response by Jeff Smith, in which he defends his article conclusions.

 

Until any of this is refuted, we've clearly established that, for relation tables:
* using a natural PK uses less space, less resources, since if you use an identity, you MUST also introduce a unique constraint on your true key columns
* using a natural PK clarifies your data model and makes it easier to read, understand and work with
* using a natural PK simplifies your SQL statements
* using a natural PK enforces data integrity without the need for an additional unique constraint
* using a natural PK results in true normalization, since your attributes are fully dependent on your primary key and not just a meaningless, random number
* using an identity PK actually complicates things like editing a relation, if you are relying only on the PK, since knowing that you are editing or viewing "CustomerProductID 3922" means nothing, while knowing that you are editing or vewing the relation between "CustomerID 23 and ProductID 4" tells you everything, simplifying your code and your logic.
I'd write more, but again, I feel as though I am just repeating the same points over and over and no one is specifically addressing them ....
Let's talk actuals, here, not theory ... can you please give me a simple example where using an identity as a PK for a relation table, instead of a composite PK, makes future changes "easier"?

 

But this got me thinking even more in the merits and shortcomings of each design technique, in primary keys and it's relation with natural keys, business keys/rules, etc.

So my response headed for evolutive maintenance, change, flexibility:

I just can't run away from a good discussion. :)

Suppose that the client comes in and wants to introduce some “minor” changes/improvements.
First, he wants to keep the history of changes in the CustomerProducts table. So you create a CustomerProductsHistory table with the timestamp of the operation.
But he also reports that querys to find associations within an OrderLimit range are running very slowly. So you add an index to the OrderLimit.

These “changes” compound and the table becomes larger and wider with more indexes. Child tables are added and because we shouldn't use a surrogate as PK, we keep the parent PK and add some other attribute. Using Ids as the PKs from Customer and Products, the effect is lessened (the size of the PK is just double of a surrogate key, but with natural keys the effect is clear), but eventually your indexes start to suffer because the PK is larger, as well as all child tables PKs and indexes.

Future changes are easier because you get more flexibility in maintenance and with the model.

  • In maintenance: if a mistake was detected and you MUST correct an association (the CustomerId or ProductId), you don’t have to mess with the PK. Because you didn’t expose the PK of the table to the customer (the surrogate key), you can change the data easily.
  • In evolutions: if there’s a need to change the CustomerProducts table (you want to turn it into a temporal table, adding a start date and end date), it’s much easier if there’s a surrogate key, especially if the table has child tables.

I think most pragmatic people, having been through similar situations, can see the benefits and flexibility of the surrogate key model. Besides that, there’s also the other advantages mentioned like:

  • Consistency in the model (easing mapping tools and development),
  • Comprehension and usage of the model (this table PK is what 4 columns? And joins this other table with what 3 columns?) because the PK is just TableNameId and joins with the FK named TableNameId

The last point I don’t understand. You say CustomerProductID 3922 means nothing. Yet CustomerId 23 has meaning? Why? Only if you look it up in the Customers table… I just hope you don’t show CustomerId 23 in your UI, because then you just exposed the PK, and you can’t ever change it easily.

I’ll concede to the argument that it’s not completely 3NF. But if you go this way, why not make all the tables 5NF mandatory?

Just my 0.02 €

And just to clear Jeff Smith's doubts:

When you create your tables, and you are adding your identity primary keys, do you also create additional unique constraints to all of your tables to ensure you have data integrity?

Got me thinking in the design of the models that I currently adhere, and why I like them:

Yes, of course.
But maybe I wasn't explicit in my last comment (or there's some ambiguity in my terms). To me TableNameId is the PK of the table. It's "private" and not exposed to the business. If the natural key (defined by the business) is just a number, I normally use TableNameNumber (eg: CustomerNumber, ProductNumber). This column is visible to the business, and follows the business rules (no gaps, prefixes, suffixes, etc).
This way the model is very consistent, predictable, flexible and usable. It may have some overhead (not much), and not be completely normalized, but to me it's a good compromise.

Marcas Technorati: ,

Wednesday, June 06, 2007

Trends in Programming Languages

It’s been a while since my last post, due to work pressure and little spare time. But in between some code analysis for the maintenance projects I’m involved, the preparation of a Linq presentation and some new blogs I discovered, I’ve been thinking about program language evolution.

It’s not new that imperative languages are integrating functional characteristics. The novelty is the big push by Microsoft, mainly Anders Helsjberg and the .NET team in that direction. Right now, we have XAML, WPF, WCF, and Linq, SilverLight, Entities, VS Orcas just ahead. I still remember the backlash that Java received because it incorporated Garbage Collection, and a Virtual Machine that run bytecode. Nowadays, the main platforms (C# and Java) all include these characteristics, and the new contenders (Ruby, Python) all incorporate it.

Recently I found a great blog that explains the characteristics and advantages of functional programming, in a way that our imperative brains can associate with. In the aptly named article Functional Programming For The Rest of Us, the author presents the following advantages:

  • Greater Expressiveness, which equates to less lines of code and less bugs;
  • Meta-programming in the same language, dismissing the need to learn a new languge and displacing many code generators;
  • Code as Data (in Lisp, it’s possible to treat code as data, by quoting), opening the door to code manipulation, analysis and generation;
  • Data as Code (in Lisp, a data structure can be “executed”);
  • Composibility, or the possibility to change/define an application behaviour by the simple composition of functions;
  • No SIDE EFFECTS
    • Easier testing as the result of a function depends only on it’s arguments and produces no side effects. This way a test with the same arguments has always the same behaviour;
    • Debugging is simpler as there are no dependencies on Global State;
    • Concurrency (with the advent of multi-cores, this is extremely important) scales much better, because locks aren’t necessary because there is no shared state;
    • Hot code deployment, without the need to persist and import state (it is passed on the stack in the arguments), it becomes possible to substitute code without stopping the system;
    • Machine assisted proofs and Optimizations, without state and side effects, it’s possible to reason about the functions and demonstrate some characteristics about it

From my recollections, this blog and some other blogs on Functional Programming, these are the characteristics attributed to Functional Programming Languages:

  • Anonymous Functions (lambda calculus)
  • Type Inference
  • Higher Order Functions (functions as a first citizen language construct. Can receive and apply functions passed as arguments)
  • Currying (adaptation to an interface, by fixing one or more arguments)
    • int pow(int i, int j);
    • square = int pow(int i, 2);
  • Lazy Evaluation allows:
    • New types of optimization (complex calculations that are not used aren't computed)
    • Creation of new abstract control structures
    • Infinite data structures
  • Continuations (allows the storing of state and the definition of return point or the function to execute)
  • Pattern Matching (definition of functions based on types, values and expressions)
  • Closures (functions that can reference the encompassing environment, typically defined in the body of another function, allowing the association of intermediate set of data with the function: http://en.wikipedia.org/wiki/Closure_%28computer_science%29)

Why does it matter or In what way does it affect me?

Due to the natural evolution of micro-processors, the future doesn't hold increases in speed but in integration of multiple cores in the same die. Because of this, it becomes very important to not limit the concurrency, which is simpler in functional programming (because of no shared state and the expressiveness).

Also the evolution in the .NET space points in the same direction (transition from imperative to declarative), with the eminent arrival of Linq. It's clear in the .NET evolution the trend to a more declarative style with the incorporation of many characteristics of Functional Programming.

.NET evolution

C# 1.0

  • Function Pointers (delegate)

C# 2.0

  • Limited Continuations (yield)
  • Closures (anonymous methods)

C# 3.0

  • Expressiviness
    • Query expressions: from x in …
    • Automatic Properties (get; set;)
    • Partial Methods (“lightweight” event handling)
    • Extension Methods: public static void Print(this string s)
  • Type Inference for local variables (var)
  • Lambda Expressions/Anonymous functions (x => x * x)
  • Anonymous Types (new {c.name, c.Phone})
  • Code as Data (ExpressionTrees)
    • Func<int, bool> nonExprLambda = x => (x & 1) == 0;
      • Initizalizes a delegate: public delegate T Func<T, A>(A a);
      • Expression<Func<int, bool>> exprLambda = x => (x & 1) == 0;
      • Is equivalent to
      • ParameterExpression xParam = Expression.Parameter(typeof(int), "x");
        Expression<Func<int, bool>> exprLambda = Expression.Lambda<Func<int, bool>>(
        Expression.EQ(
        Expression.BitAnd(xParam, Expression.Constant(1)),
        Expression.Constant(0)),
        xParam);

References

Functional Programming explained

http://www.defmacro.org/ramblings/fp.html

http://www.defmacro.org/ramblings/lisp.html

Artigo MSDN C# 3.0

http://msdn.microsoft.com/msdnmag/issues/07/06/CSharp30/default.aspx?loc=en

Expression Trees

http://www.interact-sw.co.uk/iangblog/2005/09/30/expressiontrees

http://spellcoder.com/blogs/bashmohandes/archive/2006/06/12/143.aspx

Closures

http://pluralsight.com/blogs/dbox/archive/2005/04/27/7780.aspx

http://www.claassen.net/geek/blog/2006/08/c-closures.html

Blogs

http://blogs.msdn.com/wesdyer/rss.xml

http://themechanicalbride.blogspot.com/feeds/posts/default

http://wesnerm.blogs.com/net_undocumented/index.rdf

Thursday, November 16, 2006

Enterprise Platforms

Thinking about Enterprise Platforms, I realized what a great platform SQL is. Despite all the hype around other servers, more aligned with the SOA and Web 2.0 trends, SQL Server has become a true Enterprise Platform.

It's a great data storage server, and has been constantly gaining ground on the so called big DB Servers (Oracle and IBM DB2), with features like Online Index Rebuild, Table Partitioning, Mirroring, and Snapshot Isolation. On the performance side, it swept the top TPC performance benchmarks.

But that's the beginning of the story. With SQL 2005, it gained:
  • Message Queuing with Service Broker;
  • bundled Reporting Services (not only the generation but also the delivery);
  • Notification Services (and a new mail API);
  • CLR integration (v2.0), and if the .NET Framework 3.0 is included, we even get Communications, Workflow and Presentation Foundation for free;
  • Analysis Services for BI;
  • A true Enterprise ETL tool/environment with Integration Services;

And it’s not just targeting the enterprise, with the Express edition available for free (http://msdn.microsoft.com/vstudio/express/sql/download/), everyone can have a top performing Database server (limited to 4 GB DB, 1 GB of RAM and 1 CPU) with Management Studio Express, Reporting Services and Full-Text Search.

Tuesday, October 31, 2006

Rereading No Silver Bullet

While re-reading "No Silver Bullet" (http://www-inst.eecs.berkeley.edu/~maratb/readings/NoSilverBullet.html), I was amazed by the intemporality of the article. It was published in April 1987 (Computer, Vol. 20, No. 4) and written by Frederick P. Brooks, and as it's almost TWENTY years old it feels more actual than ever.

In the article, Brooks divides the difficulties in Software Engineering between Essence and Accidental.

On the side of the essential difficulties, are the complexity, conformity, changeability and invisibility of software. Meaning that software is inherently complex as it is an abstract entity that represents complex interactions (business rules) and interfaces with complex systems/processes/procedures. Not only that, but generally it's the software that must change to accomodate the business process and frequently the client doesn't know what he wants/needs (funny to see how something never changes J).

The past breakthroughs (and some current silver bullets, in my opinion) only solve accidental difficulties. These are difficulties not inherent to building software, but with the way it was done.
Among these are: High-level Languages (raises the expressive power of the language, but doesn’t attack the complexity of software), Time-Sharing (reduces the time gap between code, build and execution) and Unified Programming Environments (back then Unix and Interlisp, J).

Among the hopes, there is ADA, OO programming, Artificial Intelligence, Graphical Programming, Environments and Tools and Workstations, all with marginal increment.

The possible breakthroughs must come from:
Buy vs Build (if one builds a system from bought components and minimal adjustments, the cost can be much lower, because it’s already tested, the delivery is immediate)
Requirements refinement and rapid prototyping (most agile methodologies revolve around the iterative construction of a system. Considering the first iterations of a system, as a prototype that evolves to the final system, this point%2