Skip to content

Conversation

hunterwilhelm
Copy link

@hunterwilhelm hunterwilhelm commented May 23, 2025

"In YouTube search for cute cats"
"In Spotify look up Discover Weekly"

Copy link
Owner

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This is not a thorough review as I don't have time for it right now, but will come back to it after I'm done with university exams.

One thing we might want to consider is to implement something like an OrConstruct that also acts as a capturing group and allows matching only known services (i.e. spotify, youtube, ...).

I would also suggest adding the "duckduckgo" service that just redirects the query to the standard search skill.

import org.stypox.dicio.sentences.Sentences.AppSearch
import org.stypox.dicio.util.StringUtils

class AppSearchSkill(correspondingSkillInfo: SkillInfo, data: StandardRecognizerData<AppSearch>)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that we need another skill, maybe it's better to extend the search skill that already exists. The search skill however has a low specificity, while this one has a high specificity at the moment (although I would suggest a medium specificity would be better), so we need to see how it plays out, maybe it is a good idea to have two separate skills.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could condense these into "in|on .app. (search for?)|lookup|(look up|for)|find .query.". You could also add another sentence with "in|on .app." at the end, the sentence recognizer should still be able to distinguish it from searching in general: "(search for?)|lookup|(look up|for)|find .query. in|on .app.".

@@ -1,4 +1,14 @@
skills:
- id: app_search
specificity: high
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
specificity: high
specificity: medium

@hunterwilhelm
Copy link
Author

Thanks for reviewing this, I'll add the changes as requested over the next week or so 🚀

Copy link
Owner

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hunterwilhelm I finally completed the review, thanks again!

One thing we might want to consider is to implement something like an OrConstruct that also acts as a capturing group and allows matching only known services (i.e. spotify, youtube, ...).

Here is how I think we should add support for this, in an even more general way. Imagine being able to assign a "label" to a part of the sentence, that gets "activated" when that part of the sentence matches with the user input. A sentence with "label"ed parts might look like "search for .query. on (YouTube:youtube)|(Spotify:spotify)", where youtube and spotify are the tags. The natural language processing module would then output a boolean value for each of the labels indicating whether it was observed or not in the user input while matching that sentence. This would enable us to parse YouTube, Spotify, and more without having to create different sentences and without having to use a generic capturing group. Futhermore, since it is a very general way to assign labels, it would possibly also be useful for other nuanced behaviors in other skills.

Here is a rough outline of how this would be implemented. If you want to do it, great, let me know if you have any questions! Otherwise let me know and I can do it myself, I understand it would be a quite deep dive into the project and would require making many choices along the way.

  • modify the sentences-compiler-plugin submodule so that it can parse a new type of captures: (e.g. call it type: bool or type: label). This would mean your - id: app \ type: string would turn into - id: youtube \ type: label \ - id: spotify \ type: label (by \ I mean newline)
  • modify dicio-sentences-compiler so that it can parse labels inside sentences. I think the :abc syntax to label the thing to the left with the "abc" label would make sense, however you would probably need to remove some old now-unused code that used : in another way (as a section name delimiter, but sections are not used by sentences-compiler-plugin anymore, though they are still in the code of dicio-sentences-compiler).
  • modify the skill submodule to add a new type of construct (e.g. call it LabelledConstruct) that just wraps another construct (any construct), and inside matchToEnd first calls the wrapped construct's matchToEnd, then checks at which places the memToEnd changed (those would be the places where the wrapped construct matched better than other stuff so far), and finally adds capture information to the StandardScore objects at those places.
  • finally modify sentences-compiler-plugin to use the new version of dicio-sentences-compiler and to generate Kotlin code that uses the newly added LabelledConstruct

Comment on lines +30 to +35
class AppSearchOutput(
private val appName: String?,
private val packageName: String?,
private val searchQuery: String?,
private val success: Boolean
) : SkillOutput {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of having a class with nullable fields, you can create a sealed interface with one class for each possible outcome. It's not so necessary, but it might make things a bit more clear. See how it's done e.g. for the Lyrics, there is just one failure mode there while here you would have three. Also, to avoid having to implement GraphicalOutput multiple times, you can make the error classes override HeadlineSpeechSkillOutput.

Comment on lines +152 to +153
<string name="skill_name_app_search">App Search</string>
<string name="skill_sentence_example_app_search">In YouTube search for cute cats</string>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<string name="skill_name_app_search">App Search</string>
<string name="skill_sentence_example_app_search">In YouTube search for cute cats</string>
<string name="skill_name_app_search">Search in apps</string>
<string name="skill_sentence_example_app_search">Search for cute cats on YouTube</string>

@hunterwilhelm
Copy link
Author

@Stypox I'll let you take this over since it's more involved. Thank you as well for the time to genuinely look at this :)
So, I'll step back and watch to see how you accomplish this. Let me know if there is anything I can do to help, whether that be smaller fixes, testing, or feedback.

I only have one thought to share, I did run into issues with "Search for cute cats on YouTube" because it would conflict with the searching on the web instead of a regular search engine. So, being able to handle prefixes and suffixes (e.g. "search ... on Youtube" and "On youtube search for ..." for the target application would be nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants