Handling temporality in human activity reasoning

Abstract

Human-aware Artificial Intelligent systems are goal directed autonomous systems that are capable of interacting, collaborating, and teaming with humans. Activity reasoning is a formal reasoning approach that aims to provide common sense reasoning capabilities to these interactive and intelligent systems. This reasoning can be done by considering evidences –which may be conflicting–related to activities a human performs. In this context, it is important to consider the temporality of such evidence in order to distinguish activities and to analyse the relations between activities. Our approach is based on formal argumentation reasoning, specifically, Timed Argumentation Frameworks (TAF), which is an appropriate technique for dealing with inconsistencies in knowledge bases. Our approach involves two steps: local selection and global selection. In the local selection, a model of the world and of the human’s mind is constructed in form of hypothetical fragments of activities (pieces of evidences) by considering a set of observations. These hypothetical fragments have two kinds of relations: a conflict relation and a temporal relation. Based on these relations, the argumentation attack notion is defined. We define two forms of attacks namely the strong and the weak attack. The former has the same characteristics of attacks in TAF whereas for the latter the TAF approach has to be extended. For determining consistent sets of hypothetical fragments, that are part of an activity or are part of a set of non-conflicting activities, extension-based argumentation semantics are applied. In the global selection, the degrees of fulfillment of activities is determined. We study some properties of our approach and apply it to a scenario where a human performs activities with different temporal relations.

1 Introduction

Human-aware Artificial Intelligent systems are goal directed autonomous systems that are capable of interacting, collaborating, and teaming with humans. The perceptions obtained by these systems can be useful for activity reasoning, intention recognition, activity verification, and activity support. This article tackles the problem of reasoning about activities a human is performing considering the temporality and durability of the activities and possible overlappings between them.

Nieves et al. [11] and Morveli-Espinoza et al. [10] used argumentation for determining inconsistent (and therefore different) activities from a set of hypothetical fragments of activities (henceforth, hypothetical fragments). In their approaches, all the perceived hypothetical fragments are analysed together as they happen at the same time and without considering their durability. However, the activities a human performs may happen at different times and have different duration, which should be reflected in the reasoning about human activity. In order to better understand the problem, let us present the following scenario. In this scenario, a man –let us call him Bob –arrives to his home after working and performs different activities until he goes to sleep (Figure 1 shows the different activities Bob performs from 17h until 6h). The activities are monitored and can be perceived in form of hypothetical fragments (we can associate an activity with the notion of goal and hypothetical fragments with sub-goals), which have to be compared in order to distinguish the activities.

Fig. 1

Bob’s activities after work from 17h until 6h.

Assume that there is a set of hypothetical fragments for the activity cooking and a set of hypothetical fragments for the activity taking a shower. Although both activities can be considered conflicting because a person cannot be frying something and at the same time taking a shower (conflicting actions), both of them were performed by Bob considering that they were performed in different time intervals. By applying the approaches proposed by Nieves et al. [11] and Morveli-Espinoza et al. [10], both sets of hypothetical fragments are analysed as they occur at the same time, which is not correct because they belong to different time intervals. The result is that only one activity could be performed, which is not correct. Thus, it is important to analyse both conflicts between activities and the temporal relation between them.

We can also notice that some activities partially (or completely) overlap, this means that there are hypothetical fragments that belong to the same time interval like the hypothetical fragments of talking with mom and cooking; however, there is no conflict between them, although they belong to different activities. In this case, another mechanism for distinguishing the activities has to be used. Let us assume that the hypothetical fragments of talking are detected at the same time of the hypothetical fragments of sleeping. This may lead to emerging conflicts since it is not normal to talk consciously when one is sleeping.

Against this background, the research questions that are addressed in this work are:

How to model the temporality constraints between human activities in the settings of activity reasoning? and

How to perform human activity reasoning considering conflicts (between actions, observations, or goals) and temporal constraints?

In addressing the first question, we will use Allen’s interval algebra [1] in order to represent the durability of the activities and the temporal relation between them. Regarding the second question, we will use hypothetical fragments for representing both an action and its context in terms of related observations (about both the human and the environment) and the goal that can be achieved by performing such action. Based on the elements of its context, emerging conflicts can be identified. We can consider hypothetical fragments as arguments. Hence, we can apply formal argumentation techniques for determining consistent sets of hypothetical fragments, which in turn will determine different activities or sets of non-conflicting activities (this is called local selection). Since temporality is taken into account we will base on Timed Abstract Framework (TAF) [4], which considers that arguments are valid during specific time intervals. Given that we also consider that attacks occur in some time intervals, we will extend TAF to support it. Besides, we propose a global selection, which aims to determine the degree of fulfillment or non-fulfillment of a given activity. Both types of selections aim to recognize the activities a human is performing.

The rest of the paper is organized as follows. Section 3 presents a short introduction about Allen’s algebra and TAF. Section 4 presents basic concepts on argumentation-based activity reasoning. Section 4 extends TAF for supporting activity reasoning and presents local selection. Section 5 presents global selection. The main properties of the proposed approach are studied in Section 6. In Section 7, we show the application of our proposal to the previously described scenario. Section 8 presents a discussion about the limitations, characteristics of the proposed approach, and compares it with some related work. Finally, Section 9 summarizes this article and outlines future research.

2 Background

In this section, we present the main concepts about the interval algebra of Allen and about the Timed Abstract Framework.

2.1 Allen’s interval algebra

Allen’s interval algebra is a calculus for temporal reasoning that was introduced by Allen [1]. It is considered a description-based approach because it associates a time interval with an occurring sub-event (part of an activity) and specifies temporal relationships among sub-events. Seven basic temporal relationships were defined: before, meets, overlaps, starts, during, finishes, and parallel. Note that before and meets describe sequential relationships while the other predicates are used to specify concurrent relationships. Table 1 shows these 13 temporal relationships (where se _i and se _j sub-events of an activity) and the notation for each relation. Each relation of the first part of the table has its respective reverse relation in the second part of the table. Only the relation parallel does not have a reverse relation.

Table 1

Allen’s temporal relationships

Relation	Notation	Diagram
se i (before) se j	se i (b)se j
se i (meets) se j	se i (m) se j
se i (overlaps) se j	se i (o) se j
se i (starts) se j	se i (s) se j
se i (during) se j	se i (d) se j
se i (finishes) se j	se i (f) se j
se i (parallel) se j	se i (p) se j
Reverse	Notation
se j (after) se i	se j (b-) se i
se j (met-by) se i	se j (m-) se i
se j (overlapped-by) se i	se j (o-) se i
se j (started-by) se i	se j (s-) se i
se j (contains) se i	se j (d-) se i
se j (finished-by) se i	se j (f-) se i

In this article, we consider hypothetical fragments (see Definition 3) as sub-events of an activity. In this sense, we will use temporal relationships to determine attacks between hypothetical fragments and to distinguish different activities.

2.2 Timed abstract framework (TAF)

The Abstract Argumentation Framework (AAF) that was introduced in the seminal paper of Dung [5] is one of the most significant developments in the computational modelling of argumentation in recent years. The AAF is composed of a set of abstract arguments and a binary relation encoding attacks between arguments. Abstract arguments may represent data, reasons, or propositions. TAF is an extension of AAFs where arguments are valid only during specific intervals of time (called availability intervals). This impacts on the attack relation, which is considered only when both the attacker and the attacked arguments are available. Thus, when identifying the set of acceptable arguments the outcome associated with a TAF may vary in time. Definitions presented in this section were extracted from [4].

A time interval is a real interval [x, y]. In order to model discontinuous periods of time, the notion of time intervals set is introduced. The set of time intervals is denoted by TDSFTDR.

The next definition extends the AAF of Dung [5] by incorporating an availability function that captures the time intervals where arguments are available.

Definition 1. (TAF) A TAF is a tuple 〈ARG, att, Av〉 where ARG is a set of arguments, att is a binary relation defined over ARG, and Av is an availability function for timed arguments, defined as Av : ARG → 2^TDSFTDR, such that Av (A) is the set of availability intervals of an argument A.

The following definitions are related to argument acceptability in TAF. Since the availability of arguments varies in time, the acceptability of a given argument A will also vary in time. Firstly, the notion of t-profile binds an argument to a set of time intervals.

Definition 2. (T-profile) Let Φ : = 〈ARG, att, Av〉 be a TAF. A timed argument profile in Φ, or just t-profile, is a pair ρ : = (A, τ) where A ∈ ARG and τ is a set of time intervals. The t-profile (A, Av (A)) is called the basic t-profile of A.

Defense is an important definition in argumentation. An argument B defenses an argument A when it attacks another argument C that attacks A. In TAF, A is defended from B when B is not available in the intervals A is available (first case). Besides, in those intervals where, although B is available, it is in turn attacked by an argument C (second case). Definition 3. (Defense) Let Φ : = 〈ARG, att, Av〉 be a TAF, $\mathcal{S}$ be a set of t-profiles, and A and B be arguments. The defense t-profile of A from B w.r.t. $\mathcal{S}$ is ${\rho }_A:=(A,{\tau }_A^B)$ , where: ${\tau }_A^B=\mathtt{Av}\left(A\right)-\mathtt{Av}\left(B\right)$ ⋃_{{(C,τ C )∈S∣(C,B)∈att}} (Av (A) ∩ Av (B) ∩ τ _C ).

Av (A) - Av (B) represents the first case, when the intervals where A is available B is not available. So, A is defended in those intervals due to the absence of B. On the other hand, (Av (A) ∩ Av (B) ∩ τ _C ) represents the second case, where in the same interval there is an argument C that defends A from B.

The following definition is about the defense profile of an argument A considering all its attacking arguments.

Definition 4.(Acceptable t-profile of A w.r.t. $\mathcal{S}$ ) Let $\mathcal{S}$ be a set of t-profiles. The acceptable t-profile for A w.r.t. $\mathcal{S}$ is ρ _A  : = (A, τ _A ), where ${\tau }_A:={\cap }_{\left\{\right(B,A)\in \mathtt{att}\}}{\tau }_A^B$ and $(A,{\tau }_A^B)$ is the defense t-profile of A from B w.r.t. $\mathcal{S}$ .

Finally, the notion of acceptability is presented.

Definition 5.(Acceptability) Let Φ : = 〈ARG, att, Av〉 be a TAF.

A set $\mathcal{S}$ of t-profiles is called t-conflict-free if there are no t-profiles (A, τ _A ), $(B,{\tau }_B)\in \mathcal{S}$ such that (A, B) ∈ att and τ _A ∩ τ _B  ≠ ∅.

A t-conflict-free set of t-profiles is a t-admissible set iff $\forall (A,{\tau }_A)\in \mathcal{S}$ it holds that (A, τ _A ) is the acceptable t-profile of A w.r.t. $\mathcal{S}$ .

A t-admissible set $\mathcal{S}$ is a t-complete extension of Φ iff $\mathcal{S}$ contains all the t-profiles that are acceptable with respect to $\mathcal{S}$ .

A set $\mathcal{S}$ is the t-grounded extension of Φ iff $\mathcal{S}$ is minimal with respect to set inclusion such that is t-admissible and t-complete.

Let cf _TAF denote the set of conflict-free sets of a TAF.

3 Building blocks

In this section, we present the building blocks definitions. These are the human activity framework and the hypothetical fragment of activity. These definitions were extracted from [11].

We start by presenting the logical language that will be used throughout the article. Let $\mathcal{L}$ be a first order logic language used to represent the mental states of the agent, ⊢ stands for the inference of classical logic, ⊤ and ⊥ denote truth and falsum respectively, and ≡ denotes classical equivalence. We use lowercase roman characters to denote atoms and uppercase Greek characters to denote formulas, such that an atomic proposition b is a formula. If b is a formula, then so is ¬b. If b and c are formulae, then so are b ∧ c, b ∨ c, and b → c. Finally, if b is a formula, then so is (b).

Next definition models the human mind. We will follow the structure of the beliefs-desires-intentions (BDI) model [2].

Definition 6.(Human Activity Framework) An human activity framework ActF is a tuple of the form $〈\mathcal{T},{\mathcal{H}}_A,\mathcal{G},\mathcal{O},\mathtt{Acts}〉$ where:

$\mathcal{T}\subseteq \mathcal{L}$ is a first order logic theory. ${\mathcal{T}}_A$ denotes the signature of $\mathcal{T}$ ;

${\mathcal{H}}_A:=\{a_1,...,a_n\}$ denotes the set of hypothetical actions that a human can perform in a world. It holds true that ${\mathcal{H}}_A\subseteq {\mathcal{T}}_A$ ;

$\mathcal{G}:=\{g_1,...,g_n\}$ denotes a set of goals of a human. It holds that $\mathcal{G}\subseteq {\mathcal{T}}_A$ ;

$\mathcal{O}:=\{o_1,...,o_n\}$ denotes a set of observation from a world. It holds that $\mathcal{O}\subseteq {\mathcal{T}}_A$ ;

Acts ⊂ 2 ^G . Acts denotes a set of activities. We assume that a set of goals defines an activity.

Besides, it holds that ${\mathcal{H}}_A,\mathcal{G}$ , and $\mathcal{O}$ are pairwise disjoint.

Given a human activity framework, one can build small pieces of knowledge which give hypothetical evidence of the achievement of a given goal by considering a set of beliefs (i.e., a set of formulas), ahypothetical action, and a set of observations of the world. These small pieces of knowledge will be called hypothetical fragments of activities:

Definition 7.(A Hypothetical Fragment of an Activity) Let ActF : = 〈T, H _A , G, O, Acts〉 be a human activity framework. A hypothetical fragment of an activity is represented by $F:=〈\mathcal{K},{\mathcal{O}}^{\prime },a,g〉$ such that:

$\mathcal{K}\subseteq \mathcal{T}$ , ${\mathcal{O}}^{\prime }\subseteq \mathcal{O}$ , $a\in {\mathcal{H}}_A$ and $g\in \mathcal{G}$ ;

$\mathcal{K}\cup {\mathcal{O}}^{\prime }\cup \left\{a\right\}$ is consistent;

$\mathcal{K}\cup {\mathcal{O}}^{\prime }\cup \left\{a\right\}⊢g$ ;

$\mathcal{K}$ and ${\mathcal{O}}^{\prime }$ are minimal w.r.t. set inclusion.

Let us denote by HF _ActF the set of hypothetical fragments that can be constructed from ActF.

4 Local selection

In this section, we introduce a framework to reasoning about human activity with respect to time. To this end, we study how to extend the TAF approach in order to support activity reasoning.

4.1 Conflict between hypothetical fragments

Let us recall that a TAF is composed of a set an arguments and of an attack binary relation defined over them. So far, we have defined hypothetical fragments, which can be seen as arguments. Hence, a definition of attack or disagreement between hypothetical fragments is still lacking. In order to define the nature of the attack, we first need to present the notion of conflict, for which, observations, actions, and goals are taken into account. Thus, a hypothetical fragment F has a conflict with a hypothetical fragment F′ when one of the following conditions hold true (i) the action of F is inconsistent with the action of F′, (ii) the goal of F is inconsistent with the goal of F′, or (iii) an observation of F is inconsistent with an observation of F′. These conflicts are formulized in the following definition.

Definition 8. (Conflict relation) Let ActF : = 〈T, H _A , G, O, Acts〉 be a human activity framework and F₁, F₂ ∈ HF _ActF be two hypothetical fragments such that $F_1=〈{\mathcal{K}}_1,{\mathcal{O}}_1^{\prime },a_1,$ g₁〉, $F_2=〈{\mathcal{K}}_2,{\mathcal{O}}_2^{\prime },a_2,g_2〉$ . F₁ has a conflict with F₂ if one of the following conditions hold:

a₁∧ a₂ ⊢ ⊥;

g₁∧ g₂ ⊢ ⊥; or

o₁∧ o₂ ⊢ ⊥.

Let conf denote the set of all the pairs of conflicting hypothetical fragments that belong to HF _ActF ; thus, (F₁, F₂) ∈ conf denote a conflict between hypothetical fragments F₁ and F₂. We can observe that conf is a symmetric relation.

Proposition 1. Let $F_1=〈{\mathcal{K}}_1,{\mathcal{O}}_1^{\prime },a_1,$ g₁〉 and $F_2=〈{\mathcal{K}}_2,{\mathcal{O}}_2^{\prime },a_2,g_2〉$ be two hypothetical fragments. If (F₁, F₂) ∈ conf, then (F₂, F₁) ∈ conf.

Proof. By reductio ad absurdum. Assume that (F₂, F₁) ∉ conf; hence, either a₁ ≡ a₂ or g₁ ≡ g₂, or o₁ ≡ o₂. This would mean that (F₁, F₂) ∉ conf, which contradicts the premise of the proposition.

□

4.2 TAF for activity reasoning

Observe that a hypothetical fragment is basically a goal-oriented action explanation which takes as input observations of the world. From an intuitive point of view, the construction of hypothetical fragments represents the process of building hypotheses about the fulfillment of some possible activities. In order to deal with the defeasible information which is present in the hypothetical fragments, we will follow a defeasible reasoning process based on attack relations between the hypothetical fragments and argumentation semantics. The output of this first selection allows to (i) know which activities can be performed in a compatible way and (ii) discern between different activities.

As we mentioned above, we will use TAF approach for dealing with defeasible information and temporality. However there is a difference in the treatment of the intervals for activity reasoning. In order to better understand, assume there are two arguments A and B whose respective intervals are [1, 4] and [2, 5]. In TAF, if there exists an attack from B to A, we say that A is attacked by B in the interval [2, 4]. However, from a hypothetical fragment view, we consider that B totally attacks A because a hypothetical fragment represents an indivisible action. This means that B attacks A in the interval [1, 5]. Since both intervals belong to [1, 5], we will consider that both hypothetical fragments are available in the same interval, called an activity interval. Thus, in this article, when we talk about t-profiles, we refer to the activity intervals.

In the activity reasoning context, the attack relation should reflect the emerging conflicts between hypothetical fragments and also consider the temporal relationship that exists between them when they belong to the same activity interval. In Section 3, we presented the seven basic temporal relationships and their corresponding converse relations. Let REL = {b, m, o, s, d, f, p, b^-, m^-, o^-, s^-, d^-, f^-} be the set of the temporal relationships. We will use function TEMPREL : HF _ActF  × HF _ActF  × TDSFTDR → 2 ^REL to associate a pair of hypothetical fragments to a type of temporal relationship.

The fact that two hypothetical fragments are conflicting is not determinant for considering that there is an attack between them. This is determined by the temporal relationship that exists between them when both of them belong to the same activity interval. Thus, when two conflicting hypothetical fragments occur in the same time interval and they have a sequential relationship (either before or meet), there is no attack between them because they can be performed in different times. On the other hand, if there is a concurrent relationship, the attack relation exists.

Definition 9.(Temporal attack relation - att) Let F, F′ be two hypothetical fragments, and x ∈ TDSFTDR a time interval such that x ∈ Av (F) and x ∈ Av (F′). We say that F′ attacks F in x when (F, F′) ∈ conf and TEMPREL (F, F′, x) = {o, s, d, f, p, o^-, s^-, d^-, f^-}.

Let (F′, F) denote that F′ attacks F.

As conf, att is also a symmetric relation.

Like hypothetical fragments, attacks also have a t-profile. In this case, it binds a pair of hypothetical fragments to a set of activity intervals where the pair of hypothetical fragments have an attack relation. Thus, a t-profile for attack is a pair ((F, F′) , τ_(F,F′)) such that (F, F′) ∈ att and τ_(F,F′) is a set of activity intervals where there is an attack between F and F′. The fact that two conflicting hypothetical fragments may or not attack in all the activity intervals they belong determines that the attack may be strong or weak.

Definition 10. (Strong and weak attacks) Let F, F′ be two hypothetical fragments; (F, Av (F)) and (F′, Av (F′)) be the t-profiles of F and F′, respectively; and τ_(F,F′) be the set of activity intervals where (F, F′) ∈ att. We say that:

Fstrongly attacksF′ when τ_(F,F′) = Av (F) ∩ Av (F′)

Fweakly attacksF′ when τ_(F,F′) ⊂ Av (F) ∩ Av (F′).

Let att _s denote the set of ordered pairs of hypothetical fragments that have strong attack relation and att _w the set of ordered pairs of hypothetical fragments that have weak attack relation. It holds that att _s ∩ att _w  = ∅. On the other hand, att = att _s  ∪ att _w denotes the attack relation in a general way without specifications.

We can now define the TAF for activity reasoning.

Definition 11. (Activity Reasoning TAF) Let ActF = 〈T, H _A , G, O, Acts〉 be a human activity framework. A TAF for activity reasoning (or simply A-TAF) is a tuple 〈HF _ActF , conf, TEMPREL, Av, att〉 where:

HF _ActF is a set of hypothetical fragments constructed from ActF;

conf ⊆ HF _ActF  × HF _ActF is a conflict relation;

TEMPREL is a function that assigns the type of temporal relation to a pair of hypothetical fragments;

Av : HF _ActF  → 2^TDSFTDR is a function that returns a set of activity intervals a hypothetical fragment holds true;

att is a set of temporal attack relations according to Definition 4.2.

In TAF approach, it is assumed that when there is an attack relation between two arguments, this holds for all the intervals where both of them belong. This is the same idea of a strong attack in A-TAF approach. However, the weak attack relation is not considered in TAF. Therefore, we need to extend the notion of defense for weak attack relation. Thus, besides considering those intervals where the attacker is not present and those intervals where there is an attacker for the attacker, we have to consider those intervals where two conflicting hypothetical fragments do not attack each other because their temporal relation is sequential. This means that when an argument and its attacker are part of the same interval but the relation is sequential there is not need of having a defending argument.

Definition 12. (Defense in A-TAF) Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be a A-TAF, $\mathcal{S}$ be a set of t-profiles, and F and F′ be two hypothetical fragments. The defense t-profile of F from F′ w.r.t. $\mathcal{S}$ is ${\rho }_F=(F,{\tau }_F^{F^{\prime }})$ , such that:

When (F′, F) ∈ att _s :

${\tau }_F^{F^{\prime }}{=}_{\mathit{def}}\mathtt{Av}\left(F\right)-\mathtt{Av}\left(F^{\prime }\right)$ ${\bigcup }_{\left\{\right(F^{″},\mathtt{Av}\left(F^{″}\right))\in \mathcal{S}\mid (F^{″},F^{\prime })\in {\mathtt{att}}_s\}}\left(\mathtt{Av}\right(F)\cap \mathtt{Av}(F^{\prime })\cap \mathtt{Av}(F^{″}\left)\right)$

When (F′, F) ∈ att _w :

${\tau }_F^{F^{\prime }}{=}_{\mathit{def}}\left(\mathtt{Av}\right(F)-\mathtt{Av}(F^{\prime }\left)\right)$ ∪ (τ_(F′,F) ∩ τ_(F″,F′)) ∪ ((Av (F) ∩ Av (F′)) - τ_(F′,F))

A t-profile of a hypothetical fragment F is considered acceptable when it defends F from all its attackers. Since ${\tau }_F^{F^{\prime }}$ is a defense t-profile already considers the type of attacks –that is, strong or weak –we do not need to redefine Definition 2.2. Next, we adapt it to the A-TAF approach.

Definition 13. (Acceptable t-profile in A-TAF) Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A-TAF, $\mathcal{S}$ be a set of hypothetical fragment t-profiles, and F ∈ HF _ActF a hypothetical fragment. The acceptable t-profile for F w.r.t. $\mathcal{S}$ is ρ _A  = (A, τ _F ), where ${\tau }_F{=}_{\mathit{def}}{\cap }_{\left\{\right(F^{\prime },F)\in \mathtt{att}\}}{\tau }_F^{F^{\prime }}$ and $(F,{\tau }_F^{F^{\prime }})$ is the defense t-profile of F from F′ w.r.t. $\mathcal{S}$ .

Based on the definition of acceptable t-profile, the notion of acceptability and semantics is constructed. In this case, we need to redefine conflict-freeness because conflict-freeness in TAF considers that when there is an attack relation between two arguments it occurs in all the intervals both of them belong. However, as in the defense case, in A-TAF two conflicting arguments can have an attacks relation in some activity intervals and not necessarily in all the activity interval they belong (weak attack).

Definition 14. (Conflict-freeness) Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A-TAF. A set $\mathcal{S}$ of hypothetical fragment t-profiles is called t-conflict-free if for all t-profile (F₁, Av (F₁) ′), $(F_2,\mathtt{Av}(F_2{)}^{\prime })\in \mathcal{S}$ , it holds true that:

(F₁, F₂) ∉ att _s and Av (F₁) ′∩ Av (F₂) ′ = ∅ or

(F₁, F₂) ∈ att _w and Av (F₁) ′ ∩ Av (F₂) ′ ⊈ τ_(F1,F2)

where Av (F₁) ′ ⊂ Av (F₁) and Av (F₂) ′ ⊂ Av (F₂).

Let cf_A-TAF denote the set of all the conflict-free sets of an A-TAF.

We can now adapt the semantics definition to the A-TAF approach. We also consider preferred semantics. This definition is adapted from [3].

Definition 15.(Semantics) Let Φ = 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF:

A t-admissible set $\mathcal{S}$ is a t-complete extension of Φ iff $\mathcal{S}$ contains all the t-profiles that are acceptable with respect to $\mathcal{S}$ .

A set $\mathcal{S}$ is the t-grounded extension of Φ iff $\mathcal{S}$ is minimal with respect to set inclusion such that is t-admissible and t-complete.

A set $\mathcal{S}$ is the t-preferred extension of Φ iff $\mathcal{S}$ is maximal with respect to set inclusion such that is t-admissible and t-complete.

An argumentation semantics SEM 1 is applied to the A-TAF in order to infer consistent sets of hypothetical fragments (extensions). An extension may contain hypothetical fragments of only one activity or hypothetical fragments of more than one activity than can be performed either at the same time without conflict or sequentially. On the other hand, we can distinguish different activities considering the conflict and the attack relations. The former can be used for activities in different activity intervals and the latter for activities in the same activity interval.

5 Global selection

Selecting hypothetical fragments by considering argumentation semantics is only one of the steps of activity recognition. An argumentation semantics can only suggest multiple competing sets of hypothetical fragments which could suggest the fulfillment of some activities. Therefore, we require a global selection of hypothetical fragments. By global selection, we mean a selection able to suggest:

degrees of both fulfillment and non-fulfillment of activities, and

evidence for believing about the fulfillment of activities.

Since activities are constituted by goals, we need to know what goals are associated to each extension. Let us recall that in an A-TAF, extensions are sets of hypothetical fragments t-profiles, so we use HYPOFRAGS (S) to denote the set of hypothetical fragments of an extension $\mathcal{S}$ . Given that a hypothetical fragment is always associated with a goal, a set of hypothetical fragments can be regarded as a set of goals. To this end, let us define the corresponding set of goals as follows: ${\mathcal{S}}^{\mathcal{G}}=\left\{g\right|〈\mathcal{K},\mathcal{O},a,g〉\in \mathtt{HYPOFRAGS}\left(\mathcal{S}\right)\}$ .

Considering that a set of hypothetical fragments can be regarded as a set of goals, the status of an activity is defined as follows:

Definition 16. (Status of Activities) Let ActF = 〈T, H _A , G, O, Acts〉 be a human activity framework, Φ = 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A-TAF with respect to ActF, and SEM be an argumentation semantics. An activity act ∈ Acts is:

achieved iff $\mathit{act}\subseteq {\mathcal{S}}^{\mathcal{G}}$ for all $\mathcal{S}\in \mathtt{SEM}\left(\Phi \right)$ .

partially-achieved iff $\exists \mathcal{S}\in \mathtt{SEM}\left(\Phi \right)$ such that $\mathit{act}\subseteq {\mathcal{S}}^{\mathcal{G}}$ and $\exists {\mathcal{S}}^{\prime }\in \mathtt{SEM}\left(\Phi \right)$ such that $\mathit{act}⊈{{\mathcal{S}}^{\prime }}^{\mathcal{G}}$

null-achieved iff for all $\mathcal{S}\in \mathtt{SEM}\left(\Phi \right)$ , $\mathit{act}⊈{\mathcal{S}}^{\mathcal{G}}$

It is important to observe that an extension $\mathcal{S}\in \mathtt{SEM}\left(\Phi \right)$ represents hypothetical fragments that argue why a particular activity is fulfilled. Considering the number of goals of each activity, we can define different degrees of achievement w.r.t each activity. Indeed, we can define a degree of achievement and a degree of non-achievement.

Definition 17.(Degree of (non-)achievement) Let ActF = 〈T, H _A , G, O, Acts〉 be a human activity framework, Φ = 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A-TAF with respect to ActF, SEM be an argumentation semantics, and act₁, act₂ ∈ Acts such that act₂ ⊆ act₁:

act₁ is (i/n)-achieved if act₂ is achieved w.r.t SEM (Φ), i = |act₂| and n = |act₁|.

act₁ is (1 - i/n)-null-achieved if act₂ is achieved w.r.t. SEM (Φ), i = |act₂| and n = |act₁|.

act₁ is (i/n)-hard-null-achieved if for all $\mathcal{E}\in \mathtt{SEM}\left(\Phi \right)$ , ${\mathit{act}}_2\cap {\mathcal{S}}^{\mathcal{G}}=\varnothing$ , i = |act₂| and n = |act₁|.

6 Properties of the approach

This section studies some properties of our approach. These properties aim to describe how the behavior of the attack relation impacts on the semantics and how, in turn, it impacts on the results of the activity reasoning.

Proposition 2 states that the resultant conflict-free sets are the same in both approaches if there are not weak attacks. This means that all the pair of attacking hypothetical fragments have a strong attack relation in A-TAF, which is equivalent to the attack between arguments in a TAF. Let us recall that the attack relation att of a TAF is equivalent to the strong attack relation att _s of a A-TAF.

Proposition 2. Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF. If att = att _s , then cf_A-TAF = cf _TAF .

Proof. By reductio ad absurdum. Assume cf_A-TAF ≠ cf _TAF . This means that $\exists \mathcal{E}\in {\mathtt{cf}}_{A-\mathit{TAF}}$ such that $\mathcal{E}\notin {\mathtt{cf}}_{\mathit{TAF}}$ . According to TAF definitions, F and F′ have an attack relation in all the intervals they belong, that is, τ_(F,F′) = Av (F) ∩ Av (F′). On the other hand, if $\mathcal{E}\in {\mathtt{cf}}_{A-\mathit{TAF}}$ , it means that there is not an attack relation between F and F′ in the activity intervals in that extension. However, according to A-TAF definitions, it does not mean that they do not attack in other activity intervals or that they never attack. In the former case, this means that τ_(F,F′) ⊂ Av (F) ∩ Av (F′), that is, (F, F′) ∈ att _w and, therefore, att ≠ att _s , which contradicts the premise of the proposition and in the latter case (F, F′) ∉ att, which also contradicts the premise of the proposition.

Proposition 3 states that when there is a weak attack relation between at least one pair of hypothetical fragments in a A-TAF, then the set of conflict-free sets in TAF is a subset of the set of conflict-free sets in A-TAF. This means that when the temporal attack relation is weak, there are more amount of consistent hypothetical fragments, that is, the person has done more activities, including those that may be conflicting. Since the attack is weak, such activities were done sequentially.

Proposition 3. Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF. If ∃ (F, F′) ∈ att _w , then cf _TAF  ⊂ cf_A-TAF.

Proof. By reductio ad absurdum. Assume that cf _TAF  = cf_A-TAF. This means that att = att _s as proved above. Hence, all (F, F′) ∈ att _s and ∄ (F, F′) ∈ att _w , which contradicts the premise of the proposition.

Finally, Proposition 4 states that when there is a weak attack relation between two hypothetical fragments, then the t-profiles of both hypothetical fragments can belong to a conflict-free set considering those activity intervals where they do not attack, that is, where they are consistent. As in Proposition 3, this means that the performed activities were sequential. On the other hand, it may happen that two conflicting hypothetical fragments are not sequential, in such case, their t-profiles cannot make part of any conflict-free set.

Proposition 4. Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF, F, F′ ∈ HF _ActF , and $\mathcal{E}\in {\mathtt{cf}}_{A-\mathit{TAF}}$ . If (F, F′) ∈ att _w and $(F,\mathtt{Av}(F{)}^{\prime }),(F^{\prime },\mathtt{Av}\left(F^{\prime }{)}^{\prime }\right)\in \mathcal{E}$ , then Av (F) ′ ∩ Av (F′) ′ ⊈ τ_(F′,F), where Av (F) ′ ⊂ Av (F) and Av (F′) ′ ⊂ Av (F′).

Proof. By reductio ad absurdum. Av (F) ′ ∩ Av (F′) ′ ⊆ τ_(F′,F). This means that there is an attack relation between F and F′ in at least one activity interval of Av (F) ′ ∩ Av (F′) ′. Hence, such activity interval (or intervals) cannot make part of the t-profile of both hypothetical fragments in the same extension. This contradicts one of the premises of the proposition.

7 Application to the scenario

In this section, we apply our proposal to the scenario presented in the introduction section. Indeed, this is a large scenario and we only take into account the necessary elements for illustrating the conflicts and attacks.

In this scenario, we focused on the temporal reasoning. For this reason we assumed that all the hypothetical fragments of each activity are known; however, it is not a rule. Indeed, formal argumentation is an appropriate technique for dealing with incomplete knowledge.

Let ActF _bob  = 〈T, H _A , G, O, Acts〉 be the Human Activity Framework for Bob’s activities, where:

$\mathcal{T}=\{r_1,r_2,r_3,r_4,r_5,r_6,r_7,r_8,r_9,r_{10},r_{11}\}$ ,

${\mathcal{H}}_A=\{a_1,a_2,a_3,a_4,a_5,a_6,a_7,a_8,a_9,a_{10},a_{11},a_{12}\}$

$\mathcal{G}=\{g_1,g_2,g_3,g_4,g_5,g_6,g_7,g_8,g_9,g_{10},g_{11},g_{12}\}$

$\mathcal{O}=\{o_1,o_2,o_3,o_4,o_5,o_6,o_7,o_8,o_9\}$

Acts = {cooking, sleeping, talking _ with _ mom, watching _ netflix, eating, taking _ shower, talk - ing, drinking} such that cooking = {g₁, g₅}, talking _ with _ mom = {g₇}, watching _ netflix = {g₁₀, g₁₁}, eating = {g₈, g₉}, taking _ shower = {g₄}, sleeping = {g₃, g₁₂} , talking = {g₂}, drinking = {g₆}.

The intended meaning of the grounded atoms and is presented below:

o1 = in _ the _ kitchen (bob)
o2 = grab (bob, pan)
o3 = move (bob, mouth)
o4 = in _ bedroom (bob)
o5 = in _ the _ bathroom (bob)
o6 = grab (bob, celular)
o7 = in _ the _ dinning (bob)
o8 = served (food)
o9 = signed _ in (netflix)
a1 = say (bob, something)

a2 = frying (bob, chicken)
a4 = lie _ down (bob, bed)
a5 = take _ shower (bob)
a6 = boil (bob, rice)
a7 = call (bob, mom)
a8 = take (bob, cutlery)
a9 = chewing (bob)
a10 = turn _ on (bob, tv)
a11 = choose (movieX)
a12 = puting _ on (pijama)
g1 = brown (chicken)
g2 = communicate (bob)
g3 = sleep (bob)
g4 = be _ clean (bob)
g5 = prepare (bob, rice)
g6 = drink (bob, coffee)
g7 = talk _ with (mom)
g8 = start (eating, bob, food)
g9 = feed (bob)
g10 = have (fun, bob)
g11 = watch (movie)
g12 = try _ to (sleep, bob)

The rules we are considering are:

r₁ = in _ the _ kitchen (X) ∧ grab (X, pan) ∧ frying (X, Y) → brown (Y)

r₂ = move (X, mouth) ∧ say (X, something) → communicate (X)

r₃ = in _ bedroom (X) ∧ has (close _ eyes, X) ∧ lie _ down (X, bed) → sleep (X)

r₄ = in _ the _ bathroom (X) ∧ take _ shower (bob) → be _ clean (X)

r₅ = in _ the _ kitchen (X) ∧ boil (X, Y) → prepare (Y)

r₆ = grab (X, celular) ∧ call (X, Y) → talk _ with (Y)

r₇ = in _ the _ dinning (X) ∧ served (Y) ∧ take (X, cutlery) → start (eating, X, Y)

r₈ = in _ the _ dinning (X) ∧ chewing (X) → feed (X)

r₉ = turn _ on (X, tv) → have (fun, X)

r₁₀ = signed _ in (netflix) ∧ choose (X) → watch (X)

r₁₁ = in _ bedroom (X) ∧ puting _ on (pijama) → try _ to (sleep, X)

r₁₂ = has (X, coffee) ∧ has (X, water) → drink (X, coffee)

From ActF _bob , the following set of hypothetical fragments can be constructed: HF _ActF  = {F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁}. Table 2 presents the details about them and Figure 2 shows these hypothetical fragments and their temporal relationship. For this example, we will work with three activity intervals: [17, 18], [18, 22], and [22, 6].

Table 2

Hypothetical fragments constructed from ActF _bob

ID	$\mathcal{K}$	${\mathcal{O}}^{\prime }$	a	g
F 1	{r5}	{o1}	a 6	g 5
F 2	{r1}	{o1, o2}	a 2	g 1
F 3	{r4}	{o5}	a 5	g 4
F 4	{r6}	{o6}	a 7	g 7
F 5	{r7}	{o7, o8}	a 8	g 8
F 6	{r8}	{o7}	a 9	g 9
F 7	{r9}	{}	a 10	g 10
F 8	{r10}	{o9}	a 11	g 11
F 9	{r11}	{o4}	a 12	g 12
F 10	{r3}	{o4}	a 4	g 3
F 11	{r2}	{o3}	a 1	g 2

Fig. 2

Recognized hypothetical fragments.

Now, let us present the emerging conflicts between the hypothetical fragments:

conf	reason	TEMPREL	att
(F1, F3)	o1 ≡ ¬ o5	b	-
(F5, F3)	o7 ≡ ¬ o5	b	-
(F1, F9)	o1 ≡ ¬ o4	b	-
(F5, F9)	o7 ≡ ¬ o4	b	-
(F1, F10)	o1 ≡ ¬ o4	b	-
(F5, F10)	o7 ≡ ¬ o5	b	-
(F2, F3)	o1 ≡ ¬ o5	b	-
(F6, F3)	o7 ≡ ¬ o5	b	-
(F2, F5)	o1 ≡ ¬ o7	m	-
(F6, F9)	o7 ≡ ¬ o4	b	-
(F2, F6)	o1 ≡ ¬ o7	b	-
(F6, F10)	o7 ≡ ¬ o4	b	-
(F2, F9)	o1 ≡ ¬ o4	b	-
(F7, F3)	a10 ≡ ¬ a5	b	-
(F2, F10)	o1 ≡ ¬ o4	b	-
(F7, F9)	g10 ≡ ¬ g12	b	-
(F3, F9)	o5 ≡ ¬ o4	m	-
(F7, F10)	g10 ≡ ¬ g3	b	-
(F3, F10)	o5 ≡ ¬ o4	b	-
(F8, F3)	a11 ≡ ¬ a5	m	-
(F4, F9)	g7 ≡ ¬ g12	b	-
(F8, F9)	g11 ≡ ¬ g12	b	-
(F4, F10)	g7 ≡ ¬ g3	b	-
(F8, F10)	g11 ≡ ¬ g3	b	-
(F9, F11)	g12 ≡ ¬ g2	b	-
(F10, F11)	g3 ≡ ¬ g2	d -	+

Let us denote with A-TAF _bob the A-TAF for reasoning about the activities of bob, The next step is to apply a semantics, in this case, the preferred semantics will be applied. The resultant preferred extensions are:

${\mathcal{S}}_1=\left\{\right(F_1,\left\{\right\}),(F_2,\left\{\right[17,18\left]\right\}),(F_4,\left\{\right[17,18\left]\right\}),(F_5,\left\{\right[18,22\left]\right\}),(F_6,\left\{\right[18,22\left]\right\}),(F_7,\left\{\right[18,22\left]\right\}),(F_8,\left\{\right[18,22\left]\right\}),(F_{11},\left\{\right[18,22\left]\right\}),(F_3,\left\{\right[22,6\left]\right\}),(F_9,\left\{\right[22,6\left]\right\}),(F_{10},\left\{\right[22,6\left]\right\}\left)\right\}$

${\mathcal{S}}_2=\left\{\right(F_1,\left\{\right[17,18\left]\right\}),(F_2,\left\{\right[17,18\left]\right\}),(F_4,\left\{\right[17,18\left]\right\}),(F_5,\left\{\right[18,22\left]\right\}),(F_6,\left\{\right[18,22\left]\right\}),(F_7,\left\{\right[18,22\left]\right\}),(F_8,\left\{\right[18,22\left]\right\}),(F_{11},\left\{\right[18,22],[22,6\left]\right\}),(F_3,\left\{\right[22,6\left]\right\}),(F_9,\left\{\right[22,6\left]\right\})$

In A-TAF _bob , we can notice that (F₁₀, F₁₁) ∈ conf and there is an attack relation between them in interval [22, 6]. For this reason, the t-profile of F₁₁ in ${\mathcal{S}}_1$ only considers that it happens in interval [18, 22]. However, the same hypothetical fragment has a different t-profile in ${\mathcal{S}}_2$ considering that it happens in intervals [18, 22] and [22, 6]. Only considering the conflict relation, these hypothetical fragments would never make part of the same extension. Nevertheless, from a temporal point of view they have an attack relation in the third activity interval; however, when F₁₁ occurs in the second interval and F₁₀ in the third one, there is not an attack relation between them.

The second step of our approach considers de global selection. The set of goals associated to each extension is the following:

${\mathcal{S}}_1^g=\{g_1,g_2,g_3,g_4,g_5,g_7,g_8,g_9,g_{10},g_{11},g_{12}\}$

${\mathcal{S}}_2^g=\{g_1,g_2,g_4,g_5,g_7,g_8,g_9,g_{10},g_{11},g_{12}\}$

Let us now present the status of the activities:

Achieved activities:cooking, talking _ with _ mom, watching _ netflix, eating, taking _ shower, and talking

Partially-achieved activities: sleeping

Null-achieved activities: drinking

Finally, let us determine the degrees of fulfillment of the activities:

cooking is 2/2-achieved w.r.t. ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$

talking _ with _ mom is 1/1-achieved w.r.t. ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$

watching _ netflix is 2/2-achieved w.r.t. ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$

eating is 2/2-achieved w.r.t. ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$

taking _ shower is 2/2-achieved w.r.t. ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$

talking is 1/1-achieved w.r.t. ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$

sleeping is 2/2-achieved w.r.t. ${\mathcal{S}}_1$ and 1/2-achieved w.r.t. ${\mathcal{S}}_2$

8 Discussion

In this section, we will discuss some aspects and limitations of our proposal. Besides, we compare it with some related work.

In this work, we have assumed that we can group hypothetical fragments in activity intervals such that these activity intervals do not overlap. This can be clearly seen in the example because a set of hypothetical fragments occur between 17 and 18 hours, the other set between 18 and 22 hours, an the last set between 22 and 6 hours. However, it may occur that there is always an overlapping between all the perceived hypothetical fragments. This is a limitation of our approach, and we plan to deal with it in future research.

The attack that emerged in the application scenario was between hypothetical fragments F₁₀ and F₁₁ that corresponded to activities sleeping and talking, respectively. After applying the semantics, we can notice that it is accepted that only one of the hypothetical fragment may happen. However, in this case it may happen that a person can talk while he is sleeping or he can stop sleeping to talk. Therefore, we need to further study this behavior and how to deal with it for improving the results.

Although the main contribution of our proposal is focused on activity reasoning, the fact of distinguishing conflicts from attacks can also be important for formal argumentation. In TAF or AAF approaches, the nature of attacks is not discussed and in structured argumentation (e.g., ASPIC+ [9], ABA [6], Delp [7]) an attack is generally related to the logical inconsistency. However, as we could analyse, in temporal activity reasoning conflicts are indeed related to logical inconsistency whereas attacks are determined using the conflicts and also the temporal relation between two hypothetical fragments. Thus, these works suggest that the nature of the attacks may depend on the contexts of the argumentation reasoning is carried out.

According to Ryoo and Aggarwal [12], statistical approaches that make use of models such as Bayesian Networks, Hidden Markov Models, and Conditional Random Fields, achieve good results with sequential activities; however, they require large learning datasets and fail to deal with temporal constraint management. Other approaches use reasoning in order to deal with temporal constraints. McKeever et al. [8] base on evidence theory to incorporate time related domain knowledge into the reasoning process. Stevenson and Dobson [13] tackle the problem of reasoning about concurrent activities. They make use of the Pyramid Match Kernel algorithm to support reasoning on recognising activities of varying grained temporal constraints. The difference with these approaches is that they do not deal with defeasible knowledge and in the former work, they do not tackle the problem of concurrent activities.

9 Conclusions and future work

This article presented an approach for activity reasoning, which uses Allen’s algebra to model the temporal constraints between hypothetical fragments of activities and extends TAF approach to support defeasible activity reasoning. The temporal and the conflict relation between hypothetical fragments determine the attack relation between them and the kind of attack relation. We demonstrated that our approach allows to distinguish between different activities by considering the activity intervals their hypothetical fragments belong.

Some future research directions were presented in previous section. Some other future research are: (i) we have used the Allen’s algebra for representing the temporal relation between hypothetical fragments. However, there are more elements of this algebra that can be used like the composition operation. It will be interesting to further study in order to determine how to apply them to activity reasoning,(ii) we plan to include uncertainty in the elements of the hypothetical fragments, and (iii) we also plan to use machine learning techniques for obtaining the necessary data for generating the hypothetical fragments; in this sense, we can complement both techniques for a better performance of the approach.