TYPO3Development » extension development

Query the Linked Data Cloud using TYPO3

Jochen Rau — Mon, 28 Mar 2011 18:10:02 +0000

The Semantic Web is coming to a TYPO3 installation near you. During the TYPO3 Transition Days 2011 in Berlin a couple of Developers—including me—discussed about how to implement some aspects of the Semantic Web Stack for TYPO3. One of the first outcome is the new Extension “semantic” providing a SPARQL Client.

You can query the Linked Data Cloud—which includes the content of Wikipedia—by specifying a Query in the SPARQL Protocol and RDF Query Language. For example, the statement to query for People born in Boston before 1900 looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

PREFIX foaf: <http: //xmlns.com/foaf/0.1/>
PREFIX dbr: <http: //dbpedia.org/resource/>
PREFIX dbo: <http: //dbpedia.org/ontology/>
PREFIX xsd: <http: //www.w3.org/2001/XMLSchema#>

SELECT ?resource ?name ?birthDate ?deathDate
WHERE {
?resource a foaf:Person .
?resource foaf:name ?name .
?resource dbo:birthPlace dbr:Boston .
?resource dbo:birthDate ?birthDate .
?resource dbo:deathDate ?deathDate .
FILTER (?birthDate < "1900-01-01"^^xsd:date)
}
ORDER BY ?deathDate

In the following video I will guide you through the installation of the Extension “semantic” and demo some of its features.

I have also recorded the first fifteen minutes in the life of the extension. In the following video you see how I kickstarted the extension. Keep in mind that the library ARC2 I used in this first step isn’t part of the extension anymore. All the code is now written from scratch, including the SPARQL Query Parser (see other posting).

If you have any specific questions about the extension or found an issue, please visit the projects page on Forge so we can keep track of it. You might also want to have a look at our roadmap or visit the Homepage of the TYPO3 community.

How to Effectively use the Repository and Query Object of Extbase?

Jochen Rau — Wed, 27 Jan 2010 17:13:32 +0000

Extbase supports very well the idea of Domain-Driven Design. It supplies a Repository class to comfortably fetch your objects from. Some handy methods are already implemented. Let’s have a look at some example code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

$blog = new Tx_BlogExample_Domain_Model_Blog('TYPO3 Development');
$blog->setDescription('A blog for professional TYPO3 developers.');

$administrator = new Tx_BlogExample_Domain_Model_Administrator();
$administrator->setName('Jochen Rau');
$administrator->setEmail('jochen.rau@example.com');
$blog->setAdministrator($administrator);

$post = new Tx_BlogExample_Domain_Model_Post('Effectively use the Query Object of Extbase');
$post->setContent('Extbase is a framework to develop ... hey, we\'re getting recursive now!');

$comment = new Tx_BlogExample_Domain_Model_Comment;
$comment->setDate(new DateTime);
$comment->setAuthor('Peter Pan');
$comment->setEmail('peter.pan@example.com');
$comment->setContent('Do you need some pills?.');
$post->addComment($comment);

$blog->addPost($post);

$blogRepository = t3lib_div::makeInstance('Tx_BlogExample_Domain_Repository_BlogRepository')
$blogRepository->add($blog);

In a first step, we create a new blog with an administrator, and a post with a comment. We instanciate the BlogRepository in line 21 and add the blog to it. That’s all we have to do. The blog is now persisted and can be fetched later from the repository again. We don’t have to say save($blog) at any time. The repository “takes care” of the blog.

Keep in mind that you have to instanciate the repository with t3lib_div::makeInstance() as it is a Singleton. If you instanciate it with new the Repository will be always an empty one if Extbase searches for objects to be persisted.

The BlogRepository class is as simple as this:

1
2
3

class Tx_BlogExample_Domain_Repository_BlogRepository
extends Tx_Extbase_Persistence_Repository {
}

Most of the methods needed are already implemented in the Tx_Extbase_Persistence_Repository. These are:

$repository->add($object)
$repository->remove($object)
$repository->replace($existingObject, $newObject)
$repository->update($modifiedObject)
$repository->findAll()
$repository->countAll() // Since Extbase 1.1
$repository->removeAll()
$repository->createQuery()
$repository->countByProperty($value) // Since Extbase 1.1
$repository->findByProperty($value)
$repository->findOneByProperty($value)

The methods add() and remove() are self-explanatory. replace() takes an untracked object and puts it at the “place” of an existing one where update() just takes an existing object and updates it with the property values of the modified object.

The findByProperty() and findOneByProperty() methods are magic methods where you can replace “Property” with the name of a property of the Blog object. For example we can write $blogRepository->findByTitle('My Blog') to fetch all Blogs with the given title. This also works for objects like in $blogRepository->findByAdministrator($administrator) but not yet for properties holding aggregates. Thus, $blogRepository->findByPost($post) won’t work by now. findOneByProperty() is pretty similar to findByProperty() but returns the only first Object found.

The countAll() and countByProperty() methods are similar to the according find methods. But the don’t build objects. They only count resulting objects.

All these methods keep the underlying storage solution (mostly a relational database) transparent to the user. But what if we have special constraints that are not covered with the methods above? Let’s say we want to find the five most recent posts of a given blog. Then we can use a Query object. Let’s ask the PostRepository for those posts:

1
2
3
4
5
6
7
8
9
10
11
12

class Tx_BlogExample_Domain_Repository_PostRepository
extends Tx_Extbase_Persistence_Repository {

public function findRecentByBlog(Tx_BlogExample_Domain_Model_Blog $blog, $limit = 5) {
$query = $this->createQuery();
$query->matching($query->equals('blog', $blog));
$query->setOrderings(array('date' => Tx_Extbase_Persistence_QueryInterface::ORDER_DESCENDING));
$query->setLimit((integer)$limit);
$posts = $query->execute();
return $posts;
}
}

The method createQuery() returns an appropriate Query object suitable for Post objects. In Line 6 we define the constraint the posts should match: $query->matching([constraint]). The constraint is returned by $query->equals('blog', $blog). There are some more methods returning a constraint:

We can define the orderings with $query->setOrderings([orderings]) where [orderings] is expected to be an array of property-order pairs. There are two constants defining the order:

Order ascending: Tx_Extbase_Persistence_QueryInterface::ORDER_ASCENDING
Order descending: Tx_Extbase_Persistence_QueryInterface::ORDER_DESCENDING

We limit the result set to a maximum of five posts with $query->setLimit((integer)$limit). And finally invoke $query->execute() which returns us the already build posts with all their comments inside (if we are not “lazy” ;-) ). There is also a method $query->count() which does not fetch the resulting posts but counts them. Note that the Query object enables chaining. Thus, the following code does the same as the given above:

1
2
3
4
5
6
7

public function findRecentByBlog(Tx_BlogExample_Domain_Model_Blog $blog, $limit = 5) {
$query = $this->createQuery();
return $query->matching($query->equals('blog', $blog))
->setOrderings(array('date' => Tx_Extbase_Persistence_QueryInterface::ORDER_DESCENDING))
->setLimit((integer)$limit)
->execute();
}

That’s it for today. I hope you got a deeper insight and I am looking forward to your comments. Maybe you want to join the discussion on the related mailing list. I have taken most of the code from the extension BlogExample which is available in the Extension Repository (TER).

A walk through the persistence layer of Extbase

Jochen Rau — Thu, 16 Jul 2009 09:34:16 +0000

Extbase is the new frame-work for TYPO3 extension developers introduced in TYPO3 v4.3. In this article I will give you a brief overview over the persistence layer of Extbase. It falls into different (sub)layers — each of the with different responsibilities. Let’s start our journey from the top level.

1. Repository

A Repository is the first objects a developer comes in contact to. It handles only domain objects and provide task oriented interface to retrieve them by saying findByFoo(‘Bar’). Most of the time you are done.

2. Query

The next level is the Query object. It provides methods to create an abstract query (abstract in sense of abstracted from where the objects come from). It has no clue about the underlying structure of the storage. It even does not now if it is based on a relational model or a tree model (like the content repository of [FLOW3|FLOW3 is a next generation web application framework built for the upcoming TYPO3 v5]). It also invokes the DataMapper (see below). An extension developer uses the Query object to create his own findByFoo() methods for his Repositories.

3. QueryObjectModel

Deeper and deeper we dig … In comparison to the Query object Query Object Model knows about the underlying concepts of relations and thus Joins and Orderings. An extension developer may use this class to specify more complex and “relational model” aware queries. That makes the creation of queries (esp. Joins) al little bit laborious. But it helps to keep abstraction.

4. StorageBackend

The Storage Backend translates the Query Object Model in syntactical correct statement (SQL). It also invokes the call to the database via a handler. An extension developer normally does not have to mak his hands dirty with SQL (which is in most cases the best way to improve performance and security).

5. DatabaseHandler

The Database Handler prepares queries, fetches the result set. It can be simply the [t3lib_db|A class in TYPO3 providing methods to access a database], or [PDO|PHP Data Objects], or your preferred solution.

6. DataMapper

And finally the Data Mapper. It maps the resulting rows on the Domain Object (it creates an empty object and fills it with the converted property values). It also resolves relations according to the loading strategy (eg. Lazy Loading Proxy).

Puh! Now we are done. Hope you enjoyed the journey.

F.A.Q

Q Are you nuts! Six different layers, dozens of objects, and bunch of new things to learn! I only wanted to say “SELECT * FROM Tx_MyExt_Foo WHERE …”!
A You are right: I am nuts (at least a litte bit ;-)). But now we are fully compatible to the (Query) API of FLOW3 and reached the best level to port Extensions forward. And I really like the Idea of having totally freedom of choice.

Q And what about performance?
A That’s indeed a very important point. We will do some extensive profiling sessions, implement Lazy Loading strategies, and strive for optimized Queries.

And please keep in mind: 90% of the work is based on the concepts and code of FLOW3. Credits goes to the FLOW3 team.

TYPO3 v5—The mental Transition

Jochen Rau — Mon, 30 Mar 2009 15:19:00 +0000

The upcoming Version 5 of TYPO3 is rewritten from scratch. It is based on a new framework called FLOW3. With TYPO3 v5 the [API|Application Programming Interface] will change completely. There are also a couple of new programming paradigms and patterns. Personally, I am very enthusiastic about the “new way”.

The past couple of months I had numerous discussions about the new version and how to migrate from TYPO3 v4 to v5. And I had to realize that there are several reservations and concerns about all the new things to learn. All of them have to be taken into account to keep the community healthy.

The people I have talked with had different backgrounds (different knowledge about TYPO3, different involvement in the community, etc.). Most of them asked one of the following questions:

When will TYPO3 v5 be available?
How to migrate content?
How to migrate code (extensions)?
How to migrate me?

Questions 1.-3. are already covered by an article of the v5 team. The most underestimated question is, how to cope with the mental transition of the developers. Many developers pointed out that they do not have enough time to dig into the new version. Maybe that is a question of how to weight the time spent on daily business and the time spent on (self) education. This prioritization has to be done not only by the developer himself but also by the company he is working for. Therefore, many of the developers decided to wait until FLOW3/TYPO3 v5 is released at a future point in time.

To enable the developers to get in touch with the paradigms of FLOW3 as soon as possible was my personal motivation to be involved in doing a back-port of the FLOW3 [MVC|MVC stands for Model-View-Controller, a way to separate different concerns of an application] frame-work. The result is ExtBase, a new frame-work for extensions and a successor of [pi_Base|The standard Plugin Base Class of TYPO3 v4]. It will be shipped with TYPO3 v4.3. It is planned to be released in May, 2009.

Furthermore, I am very happy about the decision at the TYPO3 snowboard tour to back-port Fluid, too. Fluid is a very sophisticated new templating engine mainly developed by Sebastian Kurfürst.

ExtBase in companion with Fluid encourages the developer of TYPO3 v4 based extensions to code the FLOW3 style right now. There is already an example extension called BlogExample available. At time of the release of v4.3 we also plan to give a good documentation at hand.

Although we implemented already 80% of the features there is quite a lot of work to do (e.g. implementing view helper classes, doing code reviews, and integration testing, writing documentation and tutorials). Do not hesitate to send me a message, if you want to lend us a helping hand.

Domain Driven Content Management

Jochen Rau — Sat, 07 Mar 2009 12:02:31 +0000

Domain Driven Design is—by experience—a strong concept and a good foundation to build TYPO3 extensions upon.

Domain-driven design (DDD) is an approach to the design of software, based on the two premises that complex domain designs should be based on a model, and that, for most software projects, the primary focus should be on the domain and domain logic [...].

The domain represents structure, knowledge and data. A content management system (CMS) like TYPO3 has to handle the data of the domain with respect to the domain knowledge. It aggregates the data, and visualizes it as a content of an output format.

A question emerges: How should the data stored in a domain be mapped on the content of the output? You might say: “Hey, what’s the problem? The data ‘stored’ in the extension is already the content to be displayed!”. I don’t think so. Nowadays, the domain of an extension and the domain of the CMS are woven into another. And the following questions are answered by the extension itself:

“What is the title of the article?”
“How many articles should be shown on the front-page?
“Should the subtitle of the article be emphasized?”

So, we have to spend hours and hours to adapt all the templates of an extension to fit in the overall context of a web page. That’s annoying. Now, let’s try to solve that issue. The first thing is to assign the questions to the proper domain:

“What is the title of the article?” → domain of the extension
“How many articles should be shown on the front-page? → domain of the CMS
“Should the subtitle of the article be emphasized?” → domain of the CMS

If we finished the separation of the two domains we are ready to take the next step: mapping the domain of the extension onto the output generated by the CMS. I propose a two-step process:

Map the domain objects of the extension onto the content object aggregate of the CMS
Transform the content object aggregate to the output format

The first step maps the domain objects of the extension onto content objects. These aggregated content objects represent the structure and the semantics of the output, but not its syntax (like “

The Title

”). The second step transforms the content objects to the output format. This can be done with a templating engine as usual.

The benefit of this two-step process is that the CMS “owns” the templates. So we can configure the visual representation of the domain objects of all extensions at one central place.

An Example

An example might clarify the process. Let’s assume I want to write a news extension. The first step is to design the domain of my new extension:

The domain model "News Item" of the extension

Then I have to examine, what the CMS built-in content object catalog offers to me. Hm, okay, there is already a content object called “article”. It seems to fit my purposes:

The domain model "Article" of the CMS

I configure the mapping as follows:

Author → Author
Text → Text
Title → Header
Subtitle → Subheader
Teaser → ShortDescription
PublishingDate → Date

As a result, I have a generic “Article Object” which can be added to a higher-level content object, like “Main Content Area” or “Page”. My “News Item” will be rendered depending on the configuration of the transformation process and the context the article is displayed in. I—as an extension developer—don’t have to deal with the visualization of my domain objects.

But, what if I want to get the control back over the presentation of my domain object? Then I have to define my own content object class (which can of course extend an existing content object class).

This leads to much more decoupled way of content management based on domain driven design. That’s why I called it “Domain Driven Content Management”.

I’m anxious to read your comments.

Some thoughts on extension development

Jochen Rau — Tue, 03 Mar 2009 17:02:15 +0000

Today’s development process of a TYPO3 extension comprises (usually) the following steps:

collect clients’ or personal needs
design the relational model (database design)
scaffold the extension using the kickstarter
add a lot of database queries template handling to the file _pi1, _pi2, _pi3 etc.
design the templates

This ends up in an extension design which is hard to maintain – especially for larger projects. A more modern approach uses the Model View Controller Pattern. One of the advantages of this pattern is to enable the developer to focus on the domain of the client (and thus the domain model of the extension). So, the MVC pattern is a prerequisite for Domain Driven Design. This leads to a slightly different development process:

collect clients or personal needs
design the domain model (classes) translating the domain knowledge of the client into UML
scaffolding the extension using a kickstarter
adding the business logic to the domain model classes
design the templates

What are the pros of this approach

The developer and the client use a common language.
The domain model is more decoupled from the implementation architecture.
It’s clearly defined where the business logic resides (in the domain model classes).

… and the cons

It adds complexity to the extension.
It forces the developer to learn new programing paradigms, which might be difficult to schedule in the daily work.

To be consequent we have to go one step further. The domain knowledge is captured in natural language. But natural language is much more expressive than object oriented software design in UML. As a consequence, there is a mismatch between the domain model expressed by the domain experts (the client) and the object oriented model expressed in UML.

If we want to keep all the information of the domain experts we have to define an ontology. To express the domain knowledge in an ontology, OWL seems to be a good solution to me, as it is a W3C standard and targets web applications. To edit ontologies I played around with Protégé.

In the end, we will have a process which can be named “Ontology Driven Extension Development Process”:

collect clients or personal needs
design the ontology using the same language as the client does
scaffolding the extension using a new ontology driven kickstarter
adding the business logic to the domain model classes
design the templates

What are the pros of this approach

The developer and the client use a common language.
The ontology based domain model is independent from the implementation and can be shared across different extensions with different purposes.

… and the cons

It adds complexity to the development processes.
It’s forces the developer to learn completely new paradigms an terms.

I’m looking forward to your comments an appreciate any questions.