TYPO3Development » My Projects

Query the Linked Data Cloud using TYPO3

Jochen Rau — Mon, 28 Mar 2011 18:10:02 +0000

The Semantic Web is coming to a TYPO3 installation near you. During the TYPO3 Transition Days 2011 in Berlin a couple of Developers—including me—discussed about how to implement some aspects of the Semantic Web Stack for TYPO3. One of the first outcome is the new Extension “semantic” providing a SPARQL Client.

You can query the Linked Data Cloud—which includes the content of Wikipedia—by specifying a Query in the SPARQL Protocol and RDF Query Language. For example, the statement to query for People born in Boston before 1900 looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

PREFIX foaf: <http: //xmlns.com/foaf/0.1/>
PREFIX dbr: <http: //dbpedia.org/resource/>
PREFIX dbo: <http: //dbpedia.org/ontology/>
PREFIX xsd: <http: //www.w3.org/2001/XMLSchema#>

SELECT ?resource ?name ?birthDate ?deathDate
WHERE {
?resource a foaf:Person .
?resource foaf:name ?name .
?resource dbo:birthPlace dbr:Boston .
?resource dbo:birthDate ?birthDate .
?resource dbo:deathDate ?deathDate .
FILTER (?birthDate < "1900-01-01"^^xsd:date)
}
ORDER BY ?deathDate

In the following video I will guide you through the installation of the Extension “semantic” and demo some of its features.

I have also recorded the first fifteen minutes in the life of the extension. In the following video you see how I kickstarted the extension. Keep in mind that the library ARC2 I used in this first step isn’t part of the extension anymore. All the code is now written from scratch, including the SPARQL Query Parser (see other posting).

If you have any specific questions about the extension or found an issue, please visit the projects page on Forge so we can keep track of it. You might also want to have a look at our roadmap or visit the Homepage of the TYPO3 community.

A SPARQL Query Result Parser written in PHP

Jochen Rau — Mon, 31 Jan 2011 17:21:14 +0000

Inspired by the SPARQL Query Result Parser written by Daniel Alexander Smith and published on the Enakting Blog, I just wrote an implementation in PHP. It utilizes the performant event based XML parser of PHP. In contrast to the original implementation written in Python, the current version below supports also the “datatype” and “xml:lang” attribute of literal types.

I have stripped off most of the comments and the copyright notice for better readability. You will find the full code on the project page. It also comes with unit tests.

This is just a small part of our attempt to integrate some techologies of the Semantic Web Stack into TYPO3. For the current status of the project and our future plans, please see our product backlog.

The script is part of the Semantic Web Integration project of TYPO3. You might also want to have a look at our roadmap or visit the Homepage of the TYPO3 community.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

class Tx_Semantic_Domain_Model_Sparql_QueryResultParser implements Tx_Semantic_Domain_Model_Sparql_QueryResultParserInterface {

protected $parser;
protected $results = array();
protected $currentResult = array();
protected $currentName = '';
protected $currentCharacterData = '';
protected $currentType = '';
protected $currentDataType;
protected $currentLanguage;
protected $processCharacterData = FALSE;

public function __construct() {
$this->parser = xml_parser_create();
xml_set_object($this->parser, $this);
xml_set_element_handler($this->parser, 'handleElementStart', 'handleElementStop');
xml_set_character_data_handler($this->parser, 'handleCharacterData');
}

public function __destruct() {
xml_parser_free($this->parser);
}

public function parse($document) {
$status = xml_parse($this->parser, $document);
if ($status === 1) {
return $this->results;
} else {
throw new Tx_Semantic_Domain_Model_Sparql_Exception_QueryResultParserException('Parser Error: "' . xml_error_string(xml_get_error_code($this->parser)) . '".', 1296481762);
}
}

protected function handleElementStart($parser, $elementName, $elementAttributes) {
switch ($elementName) {
case 'VARIABLE':
$this->results['variables'][] = $elementAttributes['NAME'];
$this->processCharacterData = FALSE;
break;
case 'BINDING':
$this->currentName = $elementAttributes['NAME'];
$this->processCharacterData = FALSE;
break;
case 'LITERAL':
$this->currentType = 'literal';
if (isset($elementAttributes['DATATYPE'])) {
$this->currentDatatype = $elementAttributes['DATATYPE'];
}
if (isset($elementAttributes['XML:LANG'])) {
$this->currentLanguage = $elementAttributes['XML:LANG'];
}
$this->processCharacterData = TRUE;
break;
case 'BNODE':
$this->currentType = 'bnode';
$this->processCharacterData = TRUE;
break;
case 'URI':
$this->currentType = 'uri';
$this->processCharacterData = TRUE;
break;
}
}

protected function handleElementStop($parser, $elementName) {
switch ($elementName) {
case 'BINDING':
$this->currentResult[$this->currentName] = array(
'name' => $this->currentName,
'value' => $this->currentCharacterData,
'type' => $this->currentType
);
if ($this->currentDatatype !== NULL) {
$this->currentResult[$this->currentName]['datatype'] = $this->currentDatatype;
$this->currentDatatype = NULL;
}
if ($this->currentLanguage !== NULL) {
$this->currentResult[$this->currentName]['language'] = $this->currentLanguage;
$this->currentLanguage = NULL;
}
$this->currentCharacterData = '';
break;
case 'LITERAL':
$this->processCharacterData = FALSE;
break;
case 'BNODE':
$this->processCharacterData = FALSE;
break;
case 'URI':
$this->processCharacterData = FALSE;
break;
case 'RESULT':
$this->results['results'][] = $this->currentResult;
$this->currentResult = array();
break;
}
}

protected function handleCharacterData($parser, $characterData) {
if ($this->processCharacterData === TRUE) {
$this->currentCharacterData .= $characterData;
}
}

}
?>

A book about Extbase and Fluid

Jochen Rau — Tue, 06 Jul 2010 16:22:38 +0000

Writing a book is a huge project. I never believed those people saying “Just double the time of your first estimate”. Now, after the book is finally available I would say they are wrong. You have to increase it threefold.

The book about Extbase and Fluid was written in close collaboration with Sebastian Kurfürst (core developer of Fluid) and Inken Kiupel (reader). We hope that it encourages the developers of TYPO3 extensions to get in touch with the fascinating concepts of FLOW3. And we hope that the book eases the migration to TYPO3 v5 (that’s what Extbase is all about).

We are aware of the fact that many of the English speaking TYPO3 developers are eagerly waiting for more Books about Extbase and Fluid published in English. Sebastian Kurfürst and I are currently investigating the best way to to get the book translated into English to be published on typo3.org.

If you have any questions regarding the book, or found some typos (at least 3), feel free to use the comments facility (if it is positive ;-) ) or write me an e-mail. I am really looking forward to hear from you.

You might want to support our work by ordering the book through this link.

How to Effectively use the Repository and Query Object of Extbase?

Jochen Rau — Wed, 27 Jan 2010 17:13:32 +0000

Extbase supports very well the idea of Domain-Driven Design. It supplies a Repository class to comfortably fetch your objects from. Some handy methods are already implemented. Let’s have a look at some example code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

$blog = new Tx_BlogExample_Domain_Model_Blog('TYPO3 Development');
$blog->setDescription('A blog for professional TYPO3 developers.');

$administrator = new Tx_BlogExample_Domain_Model_Administrator();
$administrator->setName('Jochen Rau');
$administrator->setEmail('jochen.rau@example.com');
$blog->setAdministrator($administrator);

$post = new Tx_BlogExample_Domain_Model_Post('Effectively use the Query Object of Extbase');
$post->setContent('Extbase is a framework to develop ... hey, we\'re getting recursive now!');

$comment = new Tx_BlogExample_Domain_Model_Comment;
$comment->setDate(new DateTime);
$comment->setAuthor('Peter Pan');
$comment->setEmail('peter.pan@example.com');
$comment->setContent('Do you need some pills?.');
$post->addComment($comment);

$blog->addPost($post);

$blogRepository = t3lib_div::makeInstance('Tx_BlogExample_Domain_Repository_BlogRepository')
$blogRepository->add($blog);

In a first step, we create a new blog with an administrator, and a post with a comment. We instanciate the BlogRepository in line 21 and add the blog to it. That’s all we have to do. The blog is now persisted and can be fetched later from the repository again. We don’t have to say save($blog) at any time. The repository “takes care” of the blog.

Keep in mind that you have to instanciate the repository with t3lib_div::makeInstance() as it is a Singleton. If you instanciate it with new the Repository will be always an empty one if Extbase searches for objects to be persisted.

The BlogRepository class is as simple as this:

1
2
3

class Tx_BlogExample_Domain_Repository_BlogRepository
extends Tx_Extbase_Persistence_Repository {
}

Most of the methods needed are already implemented in the Tx_Extbase_Persistence_Repository. These are:

$repository->add($object)
$repository->remove($object)
$repository->replace($existingObject, $newObject)
$repository->update($modifiedObject)
$repository->findAll()
$repository->countAll() // Since Extbase 1.1
$repository->removeAll()
$repository->createQuery()
$repository->countByProperty($value) // Since Extbase 1.1
$repository->findByProperty($value)
$repository->findOneByProperty($value)

The methods add() and remove() are self-explanatory. replace() takes an untracked object and puts it at the “place” of an existing one where update() just takes an existing object and updates it with the property values of the modified object.

The findByProperty() and findOneByProperty() methods are magic methods where you can replace “Property” with the name of a property of the Blog object. For example we can write $blogRepository->findByTitle('My Blog') to fetch all Blogs with the given title. This also works for objects like in $blogRepository->findByAdministrator($administrator) but not yet for properties holding aggregates. Thus, $blogRepository->findByPost($post) won’t work by now. findOneByProperty() is pretty similar to findByProperty() but returns the only first Object found.

The countAll() and countByProperty() methods are similar to the according find methods. But the don’t build objects. They only count resulting objects.

All these methods keep the underlying storage solution (mostly a relational database) transparent to the user. But what if we have special constraints that are not covered with the methods above? Let’s say we want to find the five most recent posts of a given blog. Then we can use a Query object. Let’s ask the PostRepository for those posts:

1
2
3
4
5
6
7
8
9
10
11
12

class Tx_BlogExample_Domain_Repository_PostRepository
extends Tx_Extbase_Persistence_Repository {

public function findRecentByBlog(Tx_BlogExample_Domain_Model_Blog $blog, $limit = 5) {
$query = $this->createQuery();
$query->matching($query->equals('blog', $blog));
$query->setOrderings(array('date' => Tx_Extbase_Persistence_QueryInterface::ORDER_DESCENDING));
$query->setLimit((integer)$limit);
$posts = $query->execute();
return $posts;
}
}

The method createQuery() returns an appropriate Query object suitable for Post objects. In Line 6 we define the constraint the posts should match: $query->matching([constraint]). The constraint is returned by $query->equals('blog', $blog). There are some more methods returning a constraint:

$query->logicalAnd($constraint1, $constraint2);
$query->logicalOr($constraint1, $constraint2);
$query->logicalNot($constraint);
$query->withUid($uid);
$query->equals($propertyName, $operand, $caseSensitive = TRUE);
$query->in($propertyName, $operand); // Since Extbase 1.1
$query->contains($propertyName, $operand); // Since Extbase 1.1
$query->like($propertyName, $operand);
$query->lessThan($propertyName, $operand);
$query->lessThanOrEqual($propertyName, $operand);
$query->greaterThan($propertyName, $operand);
$query->greaterThanOrEqual($propertyName, $operand);

We can define the orderings with $query->setOrderings([orderings]) where [orderings] is expected to be an array of property-order pairs. There are two constants defining the order:

Order ascending: Tx_Extbase_Persistence_QueryInterface::ORDER_ASCENDING
Order descending: Tx_Extbase_Persistence_QueryInterface::ORDER_DESCENDING

We limit the result set to a maximum of five posts with $query->setLimit((integer)$limit). And finally invoke $query->execute() which returns us the already build posts with all their comments inside (if we are not “lazy” ;-) ). There is also a method $query->count() which does not fetch the resulting posts but counts them. Note that the Query object enables chaining. Thus, the following code does the same as the given above:

1
2
3
4
5
6
7

public function findRecentByBlog(Tx_BlogExample_Domain_Model_Blog $blog, $limit = 5) {
$query = $this->createQuery();
return $query->matching($query->equals('blog', $blog))
->setOrderings(array('date' => Tx_Extbase_Persistence_QueryInterface::ORDER_DESCENDING))
->setLimit((integer)$limit)
->execute();
}

That’s it for today. I hope you got a deeper insight and I am looking forward to your comments. Maybe you want to join the discussion on the related mailing list. I have taken most of the code from the extension BlogExample which is available in the Extension Repository (TER).

A walk through the persistence layer of Extbase

Jochen Rau — Thu, 16 Jul 2009 09:34:16 +0000

Extbase is the new frame-work for TYPO3 extension developers introduced in TYPO3 v4.3. In this article I will give you a brief overview over the persistence layer of Extbase. It falls into different (sub)layers — each of the with different responsibilities. Let’s start our journey from the top level.

1. Repository

A Repository is the first objects a developer comes in contact to. It handles only domain objects and provide task oriented interface to retrieve them by saying findByFoo(‘Bar’). Most of the time you are done.

2. Query

The next level is the Query object. It provides methods to create an abstract query (abstract in sense of abstracted from where the objects come from). It has no clue about the underlying structure of the storage. It even does not now if it is based on a relational model or a tree model (like the content repository of [FLOW3|FLOW3 is a next generation web application framework built for the upcoming TYPO3 v5]). It also invokes the DataMapper (see below). An extension developer uses the Query object to create his own findByFoo() methods for his Repositories.

3. QueryObjectModel

Deeper and deeper we dig … In comparison to the Query object Query Object Model knows about the underlying concepts of relations and thus Joins and Orderings. An extension developer may use this class to specify more complex and “relational model” aware queries. That makes the creation of queries (esp. Joins) al little bit laborious. But it helps to keep abstraction.

4. StorageBackend

The Storage Backend translates the Query Object Model in syntactical correct statement (SQL). It also invokes the call to the database via a handler. An extension developer normally does not have to mak his hands dirty with SQL (which is in most cases the best way to improve performance and security).

5. DatabaseHandler

The Database Handler prepares queries, fetches the result set. It can be simply the [t3lib_db|A class in TYPO3 providing methods to access a database], or [PDO|PHP Data Objects], or your preferred solution.

6. DataMapper

And finally the Data Mapper. It maps the resulting rows on the Domain Object (it creates an empty object and fills it with the converted property values). It also resolves relations according to the loading strategy (eg. Lazy Loading Proxy).

Puh! Now we are done. Hope you enjoyed the journey.

F.A.Q

Q Are you nuts! Six different layers, dozens of objects, and bunch of new things to learn! I only wanted to say “SELECT * FROM Tx_MyExt_Foo WHERE …”!
A You are right: I am nuts (at least a litte bit ;-)). But now we are fully compatible to the (Query) API of FLOW3 and reached the best level to port Extensions forward. And I really like the Idea of having totally freedom of choice.

Q And what about performance?
A That’s indeed a very important point. We will do some extensive profiling sessions, implement Lazy Loading strategies, and strive for optimized Queries.

And please keep in mind: 90% of the work is based on the concepts and code of FLOW3. Credits goes to the FLOW3 team.